[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:10 dist-test-slave-8sjz krb5kdc[11316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:10 dist-test-slave-8sjz krb5kdc[11316](info): set up 2 sockets
Sep 28 12:58:10 dist-test-slave-8sjz krb5kdc[11316](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 12:58:11.402940 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:11.527216 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34261
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:34261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:11.611800 11329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:11.612089 11329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:11.612138 11329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:11.613350 11329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:11.613428 11329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:11.613454 11329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:11.613478 11329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:11.613507 11329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:11.615095 11329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:34261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:11.615420 11329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:11.615680 11329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:11.621217 11337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:11.621333 11329 server_base.cc:1034] running on GCE node
W20260928 12:58:11.622897 11335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:11.622931 11334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:11.623265 11329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:11.623597 11329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:11.624776 11329 hybrid_clock.cc:648] HybridClock initialized: now 1790600291624760 us; error 39 us; skew 500 ppm
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:11.627300 11329 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:11.627820 11329 webserver.cc:460] Webserver started at http://127.11.11.126:43401/ using document root <none> and password file <none>
I20260928 12:58:11.628103 11329 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:11.628175 11329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:11.628317 11329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:11.629456 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "545adf87e8714b72844c803da577a6e9"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "b437233b3242ca78e790c3318b2d8035"
server_key_iv: "83f04be41bf3965702e59e70b86ec5a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.629897 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "545adf87e8714b72844c803da577a6e9"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "b437233b3242ca78e790c3318b2d8035"
server_key_iv: "83f04be41bf3965702e59e70b86ec5a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.631474 11329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:11.632323 11344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:11.632489 11329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:11.632599 11329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "545adf87e8714b72844c803da577a6e9"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "b437233b3242ca78e790c3318b2d8035"
server_key_iv: "83f04be41bf3965702e59e70b86ec5a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.632722 11329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:11.658093 11329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:11.660604 11329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:11.660780 11329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:11.664871 11329 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:34261
I20260928 12:58:11.664912 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:34261 every 8 connection(s)
I20260928 12:58:11.665307 11329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:11.666182 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:11.668637 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: Bootstrap starting.
I20260928 12:58:11.669276 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:11.669572 11397 log.cc:824] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:11.670279 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: No bootstrap required, opened a new log
I20260928 12:58:11.671895 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } }
I20260928 12:58:11.672057 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:11.672241 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545adf87e8714b72844c803da577a6e9, State: Initialized, Role: FOLLOWER
I20260928 12:58:11.672462 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } }
I20260928 12:58:11.672540 11397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:11.672582 11397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:11.672636 11397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:11.673213 11397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } }
I20260928 12:58:11.673301 11397 leader_election.cc:302] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545adf87e8714b72844c803da577a6e9; no voters:
I20260928 12:58:11.673414 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11329
I20260928 12:58:11.673496 11397 leader_election.cc:288] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:11.673530 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:11.673516 11400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:11.673731 11400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [term 1 LEADER]: Becoming Leader. State: Replica: 545adf87e8714b72844c803da577a6e9, State: Running, Role: LEADER
I20260928 12:58:11.673869 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:11.673861 11400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } }
I20260928 12:58:11.674297 11309 external_mini_cluster.cc:1428] Setting key 9e1d09111868e052cdbae91ba107aa1f
I20260928 12:58:11.674516 11401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545adf87e8714b72844c803da577a6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } } }
I20260928 12:58:11.674607 11401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:11.674772 11402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545adf87e8714b72844c803da577a6e9. Latest consensus state: current_term: 1 leader_uuid: "545adf87e8714b72844c803da577a6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545adf87e8714b72844c803da577a6e9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34261 } } }
I20260928 12:58:11.674818 11402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9 [sys.catalog]: This master's current role is: LEADER
W20260928 12:58:11.675958 11416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:11.676021 11416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:11.676102 11409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:11.676271 11409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:11.677881 11409 catalog_manager.cc:1347] Generated new cluster ID: 47b6f1acc05a440db17820a5e099ba79
I20260928 12:58:11.677934 11409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:11.687099 11409 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:11.687817 11409 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:11.690785 11409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545adf87e8714b72844c803da577a6e9: Generated new TSK 0
I20260928 12:58:11.690934 11409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:11.693382 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:11.675234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.11.11.126:34261)
0928 12:58:11.675404 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:11.675414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:11.676103 (+ 689us) server_negotiation.cc:407] Connection header received
0928 12:58:11.678254 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:11.678261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:11.678319 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:11.678478 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:11.680521 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.682202 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.683767 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.684522 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.684647 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:11.689173 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:11.689178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:11.689179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:11.689196 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:11.691290 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:11.691814 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:11.691817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:11.691817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:11.691932 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:11.692283 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:11.692284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:11.692284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:11.692503 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:11.692642 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:11.693029 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 12:58:11.693144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:11.719383 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:11.802224 11423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:11.802461 11423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:11.802488 11423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:11.803802 11423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:11.803848 11423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:11.803900 11423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:11.805425 11423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:11.805691 11423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:11.805943 11423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:11.811075 11431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:11.811235 11423 server_base.cc:1034] running on GCE node
W20260928 12:58:11.811182 11428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:11.811089 11429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:11.811506 11423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:11.811810 11423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:11.812981 11423 hybrid_clock.cc:648] HybridClock initialized: now 1790600291812967 us; error 42 us; skew 500 ppm
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:11.815517 11423 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:11.816092 11423 webserver.cc:460] Webserver started at http://127.11.11.65:35913/ using document root <none> and password file <none>
I20260928 12:58:11.816335 11423 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:11.816406 11423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:11.816566 11423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:11.817646 11423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "4c6faac6db3b445d93f3ead90dfee9cc"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "abe02d361c0b5f3678508267d8c7a864"
server_key_iv: "38101d7d688b15acebf28cd9981307ae"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.818099 11423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "4c6faac6db3b445d93f3ead90dfee9cc"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "abe02d361c0b5f3678508267d8c7a864"
server_key_iv: "38101d7d688b15acebf28cd9981307ae"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.819540 11423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:11.820346 11438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:11.820513 11423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:11.820619 11423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "4c6faac6db3b445d93f3ead90dfee9cc"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "abe02d361c0b5f3678508267d8c7a864"
server_key_iv: "38101d7d688b15acebf28cd9981307ae"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.820694 11423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:11.827628 11423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:11.829687 11423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:11.829866 11423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:11.830094 11423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:11.830406 11423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:11.830461 11423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:11.830502 11423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:11.830530 11423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:11.836640 11423 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:40777
I20260928 12:58:11.836683 11551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:40777 every 8 connection(s)
I20260928 12:58:11.837018 11423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:11.844185 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11423
I20260928 12:58:11.844301 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:11.844455 11309 external_mini_cluster.cc:1428] Setting key 81ca071c3621751c527aa84df2ed824e
I20260928 12:58:11.847628 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:11.837795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59627 (local address 127.11.11.126:34261)
0928 12:58:11.837858 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:11.837860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:11.838060 (+ 200us) server_negotiation.cc:407] Connection header received
0928 12:58:11.838541 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:11.838542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:11.838552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:11.838617 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:11.839642 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.840188 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.841143 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.841343 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.841399 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:11.843788 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:11.843789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:11.843789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:11.843796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:11.845646 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:11.846244 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:11.846245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:11.846246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:11.846335 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:11.846692 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:11.846694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:11.846695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:11.846863 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:11.846918 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:11.847364 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 12:58:11.847517 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:11.847770 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:11.837773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34261 (local address 127.11.11.65:59627)
0928 12:58:11.837928 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:11.837971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:11.838459 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:11.838674 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:11.838677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:11.839342 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:11.839544 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:11.839549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.840266 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.840267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:11.841065 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:11.841067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:11.841446 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:11.841446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:11.841567 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:11.842149 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:11.842160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:11.843680 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:11.845735 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:11.845742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:11.845760 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:11.846102 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:11.846404 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:11.846405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:11.846406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:11.846612 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:11.846973 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:11.846980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:11.847181 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 12:58:11.847334 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 12:58:11.847520 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 12:58:11.848702 11552 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34261
I20260928 12:58:11.848842 11552 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:11.849036 11552 heartbeater.cc:511] Master 127.11.11.126:34261 requested a full tablet report, sending...
I20260928 12:58:11.849597 11361 ts_manager.cc:194] Registered new tserver with Master: 4c6faac6db3b445d93f3ead90dfee9cc (127.11.11.65:40777)
I20260928 12:58:11.850310 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:59627
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:11.871378 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:11.968719 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:11.969000 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:11.969050 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:11.970502 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:11.970577 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:11.970626 11557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:11.972311 11557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:11.972596 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:11.972846 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:11.978118 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:11.978124 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:11.978233 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:11.978317 11557 server_base.cc:1034] running on GCE node
I20260928 12:58:11.978523 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:11.978731 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:11.979908 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790600291979895 us; error 32 us; skew 500 ppm
Sep 28 12:58:11 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:11.982533 11557 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:11.983084 11557 webserver.cc:460] Webserver started at http://127.11.11.66:42127/ using document root <none> and password file <none>
I20260928 12:58:11.983354 11557 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:11.983425 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:11.983577 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:11.984797 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "681752742cd8414e8bfb2d13ecfb5f82"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "de15dbe6d1ed9af0f6ff04c20aad8c4b"
server_key_iv: "b7cdfad6c14de6541aadf7681e92cd05"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.985167 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "681752742cd8414e8bfb2d13ecfb5f82"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "de15dbe6d1ed9af0f6ff04c20aad8c4b"
server_key_iv: "b7cdfad6c14de6541aadf7681e92cd05"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.986657 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:11.987437 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:11.987612 11557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:11.987720 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "681752742cd8414e8bfb2d13ecfb5f82"
format_stamp: "Formatted at 2026-09-28 12:58:11 on dist-test-slave-8sjz"
server_key: "de15dbe6d1ed9af0f6ff04c20aad8c4b"
server_key_iv: "b7cdfad6c14de6541aadf7681e92cd05"
server_key_version: "encryptionkey@0"
I20260928 12:58:11.987803 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:11.998415 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:12.000999 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:12.001185 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:12.001438 11557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:12.001773 11557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:12.001828 11557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:12.001869 11557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:12.001899 11557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:12.007933 11557 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:38145
I20260928 12:58:12.007982 11685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:38145 every 8 connection(s)
I20260928 12:58:12.008369 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:12.016662 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11557
I20260928 12:58:12.016801 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:12.016989 11309 external_mini_cluster.cc:1428] Setting key f43ff1ccfbc7b0dadcd52ee82087a661
I20260928 12:58:12.018384 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.009084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:34321 (local address 127.11.11.126:34261)
0928 12:58:12.009127 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:12.009129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:12.009368 (+ 239us) server_negotiation.cc:407] Connection header received
0928 12:58:12.009882 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:12.009884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:12.009898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:12.009967 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:12.010939 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.011451 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.012539 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.012734 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.012804 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.015104 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:12.015106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:12.015106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:12.015113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:12.016767 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.017290 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.017291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.017291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.017364 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.017696 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.017697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.017698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.017827 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:12.017884 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:12.018241 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 12:58:12.018320 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:12.018725 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.009080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34261 (local address 127.11.11.66:34321)
0928 12:58:12.009232 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:12.009268 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:12.009748 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:12.010024 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:12.010029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:12.010631 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:12.010841 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:12.010846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.011536 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.011537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:12.012461 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:12.012464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.012842 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.012844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:12.012963 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.013590 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:12.013599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:12.015006 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:12.016858 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:12.016865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:12.016867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:12.017129 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:12.017444 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:12.017445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:12.017445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:12.017615 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:12.017899 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:12.017907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:12.018096 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 12:58:12.018221 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 12:58:12.018425 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260928 12:58:12.019281 11686 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34261
I20260928 12:58:12.019392 11686 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:12.019584 11686 heartbeater.cc:511] Master 127.11.11.126:34261 requested a full tablet report, sending...
I20260928 12:58:12.020022 11361 ts_manager.cc:194] Registered new tserver with Master: 681752742cd8414e8bfb2d13ecfb5f82 (127.11.11.66:38145)
I20260928 12:58:12.020550 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:34321
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:12.043430 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:12.160264 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:12.160527 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:12.160573 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:12.162016 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:12.162083 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:12.162132 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:12.163838 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:12.164147 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:12.164395 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:12.171028 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:12.171146 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:12.171033 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:12.171279 11691 server_base.cc:1034] running on GCE node
I20260928 12:58:12.171507 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:12.171770 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:12.172952 11691 hybrid_clock.cc:648] HybridClock initialized: now 1790600292172935 us; error 28 us; skew 500 ppm
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:12.175959 11691 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:12.176487 11691 webserver.cc:460] Webserver started at http://127.11.11.67:37383/ using document root <none> and password file <none>
I20260928 12:58:12.176689 11691 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:12.176731 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:12.176828 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:12.177873 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "3bb5978850f7462baabc1dd8c6c67dc7"
format_stamp: "Formatted at 2026-09-28 12:58:12 on dist-test-slave-8sjz"
server_key: "4751faae235a07971d49cf33fa27058b"
server_key_iv: "5cec6256707dcd945256a254796f8aec"
server_key_version: "encryptionkey@0"
I20260928 12:58:12.178239 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "3bb5978850f7462baabc1dd8c6c67dc7"
format_stamp: "Formatted at 2026-09-28 12:58:12 on dist-test-slave-8sjz"
server_key: "4751faae235a07971d49cf33fa27058b"
server_key_iv: "5cec6256707dcd945256a254796f8aec"
server_key_version: "encryptionkey@0"
I20260928 12:58:12.179911 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:12.180809 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:12.180998 11691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:12.181105 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "3bb5978850f7462baabc1dd8c6c67dc7"
format_stamp: "Formatted at 2026-09-28 12:58:12 on dist-test-slave-8sjz"
server_key: "4751faae235a07971d49cf33fa27058b"
server_key_iv: "5cec6256707dcd945256a254796f8aec"
server_key_version: "encryptionkey@0"
I20260928 12:58:12.181193 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:12.191529 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:12.193989 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:12.194168 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:12.194410 11691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:12.194737 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:12.194792 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:12.194833 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:12.194860 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:12.209522 11691 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:38473
I20260928 12:58:12.209657 11819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:38473 every 8 connection(s)
I20260928 12:58:12.210027 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:12.219533 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11691
I20260928 12:58:12.219633 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:12.219801 11309 external_mini_cluster.cc:1428] Setting key 6d7bd08409702dbd3763e519d00d2fa1
I20260928 12:58:12.221731 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.210882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:40661 (local address 127.11.11.126:34261)
0928 12:58:12.210927 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:12.210929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:12.211357 (+ 428us) server_negotiation.cc:407] Connection header received
0928 12:58:12.212240 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:12.212242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:12.212254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:12.212310 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:12.213505 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.214016 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.215218 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.215420 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.215498 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.218309 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:12.218311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:12.218311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:12.218318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:12.220105 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.220631 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.220632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.220633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.220707 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.221005 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.221006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.221006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.221154 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:12.221211 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:12.221554 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 12:58:12.221633 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:12.221948 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.210883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34261 (local address 127.11.11.67:40661)
0928 12:58:12.211202 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:12.211241 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:12.212121 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:12.212388 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:12.212393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:12.213054 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:12.213380 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:12.213386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.214130 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.214131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:12.215110 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:12.215112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.215504 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.215505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:12.215685 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.216368 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:12.216377 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:12.218212 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:12.220201 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:12.220208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:12.220211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:12.220559 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:12.220788 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:12.220788 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:12.220789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:12.220936 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:12.221252 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:12.221257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:12.221386 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 12:58:12.221494 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:58:12.221625 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":174,"threads_started":1}
I20260928 12:58:12.222484 11820 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34261
I20260928 12:58:12.222596 11820 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:12.222841 11820 heartbeater.cc:511] Master 127.11.11.126:34261 requested a full tablet report, sending...
I20260928 12:58:12.223330 11361 ts_manager.cc:194] Registered new tserver with Master: 3bb5978850f7462baabc1dd8c6c67dc7 (127.11.11.67:38473)
I20260928 12:58:12.223763 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:40661
I20260928 12:58:12.232096 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:12.258641 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.249478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.11.11.65:40777)
0928 12:58:12.249675 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:12.249711 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:12.249723 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:12.249751 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:12.249753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:12.249768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:12.249842 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:12.250571 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.251331 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.252227 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.252562 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.252622 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.255339 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:12.255349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:12.255350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:12.255357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:12.257251 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.257560 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.257563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.257563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.257618 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.257875 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.257876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.257877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.258036 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:12.258263 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:12.258302 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:58:12.258424 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":124,"threads_started":1}
W20260928 12:58:12.259063 11466 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59704
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:12.283977 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:12.275784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.11.11.65:40777)
0928 12:58:12.275849 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:12.275851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:12.275897 (+ 46us) server_negotiation.cc:407] Connection header received
0928 12:58:12.275954 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:12.275954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:12.275967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:12.276017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:12.276820 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.277410 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.278173 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:12.278433 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:12.278491 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:12.280936 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:12.280937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:12.280937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:12.280945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:12.282783 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.283129 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.283130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.283131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.283194 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:12.283407 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:12.283408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:12.283409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:12.283521 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:12.283674 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:12.283765 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:58:12.283852 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:58:12.284703 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11423
I20260928 12:58:12.290045 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11557
I20260928 12:58:12.295982 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11691
I20260928 12:58:12.300590 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11329
2026-09-28T12:58:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1954 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11834](info): set up 2 sockets
Sep 28 12:58:12 dist-test-slave-8sjz krb5kdc[11834](info): commencing operation
krb5kdc: starting...
W20260928 12:58:13.332567 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:13.455072 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43177
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:13.536753 11847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:13.536962 11847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:13.536984 11847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:13.538067 11847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:13.538100 11847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:13.538115 11847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:13.538126 11847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:13.538138 11847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:13.539531 11847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:43177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:13.539738 11847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:13.539906 11847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:13.544790 11853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:13.544746 11852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:13.544871 11855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:13.545220 11847 server_base.cc:1034] running on GCE node
I20260928 12:58:13.545478 11847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:13.545768 11847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:13.546939 11847 hybrid_clock.cc:648] HybridClock initialized: now 1790600293546926 us; error 43 us; skew 500 ppm
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:13.549619 11847 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:13.550160 11847 webserver.cc:460] Webserver started at http://127.11.11.126:37575/ using document root <none> and password file <none>
I20260928 12:58:13.550395 11847 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:13.550462 11847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:13.550613 11847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:13.551690 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "baf7bbf36e5575ed27304a5197dfcb92"
server_key_iv: "ee91b057b92957d512bee871baa63108"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.552115 11847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "baf7bbf36e5575ed27304a5197dfcb92"
server_key_iv: "ee91b057b92957d512bee871baa63108"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.553552 11847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:13.554297 11862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.554497 11847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:13.554628 11847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "baf7bbf36e5575ed27304a5197dfcb92"
server_key_iv: "ee91b057b92957d512bee871baa63108"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.554716 11847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:13.565774 11847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:13.568342 11847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:13.568511 11847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:13.572500 11847 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:43177
I20260928 12:58:13.572566 11914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:43177 every 8 connection(s)
I20260928 12:58:13.572948 11847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:13.573879 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:13.576265 11915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: Bootstrap starting.
I20260928 12:58:13.576872 11915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:13.577185 11915 log.cc:824] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:13.577943 11915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: No bootstrap required, opened a new log
I20260928 12:58:13.579668 11915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } }
I20260928 12:58:13.579806 11915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:13.579829 11915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae90e9ec4ef4d8d8183bc60b1f4298c, State: Initialized, Role: FOLLOWER
I20260928 12:58:13.580068 11915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } }
I20260928 12:58:13.580184 11915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:13.580173 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11847
I20260928 12:58:13.580231 11915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:13.580273 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:13.580286 11915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:13.580435 11309 external_mini_cluster.cc:1428] Setting key 90dd91d9447f5fc70d1a607bbdf5e1b8
I20260928 12:58:13.581085 11915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } }
I20260928 12:58:13.581218 11915 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae90e9ec4ef4d8d8183bc60b1f4298c; no voters:
I20260928 12:58:13.581393 11915 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:13.581423 11920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:13.581571 11920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [term 1 LEADER]: Becoming Leader. State: Replica: 9ae90e9ec4ef4d8d8183bc60b1f4298c, State: Running, Role: LEADER
I20260928 12:58:13.581673 11915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:13.581670 11920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } }
I20260928 12:58:13.583195 11922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae90e9ec4ef4d8d8183bc60b1f4298c. Latest consensus state: current_term: 1 leader_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } } }
I20260928 12:58:13.583240 11921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae90e9ec4ef4d8d8183bc60b1f4298c" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43177 } } }
I20260928 12:58:13.583520 11921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:13.583508 11922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c [sys.catalog]: This master's current role is: LEADER
W20260928 12:58:13.583871 11934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:13.583930 11934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:13.584446 11935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:13.584637 11935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:13.586608 11935 catalog_manager.cc:1347] Generated new cluster ID: 5d1d94acadee4838bffdd4883d554dc0
I20260928 12:58:13.586689 11935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:13.592885 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:13.581031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.11.11.126:43177)
0928 12:58:13.581258 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:13.581268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:13.581300 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:58:13.581425 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:13.581433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:13.581495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:13.582512 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:13.582632 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.584221 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.585329 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.585914 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.586038 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:13.588868 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:13.588873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:13.588874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:13.588887 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:13.591053 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.591493 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.591497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.591498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.591630 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.591845 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.591846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.591846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.592218 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:13.592407 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:13.592482 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:58:13.592656 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260928 12:58:13.597503 11935 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:13.598304 11935 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:13.601949 11935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae90e9ec4ef4d8d8183bc60b1f4298c: Generated new TSK 0
I20260928 12:58:13.602175 11935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:13.635804 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:13.739403 11941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:13.739701 11941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:13.739748 11941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:13.741537 11941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:13.741600 11941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:13.741632 11941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:13.743328 11941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:13.743626 11941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:13.743880 11941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:13.748943 11947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:13.749028 11946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:13.748943 11949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:13.749223 11941 server_base.cc:1034] running on GCE node
I20260928 12:58:13.749462 11941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:13.749714 11941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:13.750877 11941 hybrid_clock.cc:648] HybridClock initialized: now 1790600293750863 us; error 45 us; skew 500 ppm
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:13.753512 11941 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:13.754031 11941 webserver.cc:460] Webserver started at http://127.11.11.65:45081/ using document root <none> and password file <none>
I20260928 12:58:13.754236 11941 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:13.754285 11941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:13.754387 11941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:13.755452 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "01c022b47f4021d5748a6aa797e612d8"
server_key_iv: "f15148922db6357f55d34c663b08ea9d"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.755761 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "01c022b47f4021d5748a6aa797e612d8"
server_key_iv: "f15148922db6357f55d34c663b08ea9d"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.757205 11941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:13.758041 11956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.758244 11941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:13.758337 11941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "01c022b47f4021d5748a6aa797e612d8"
server_key_iv: "f15148922db6357f55d34c663b08ea9d"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.758419 11941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:13.781668 11941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:13.784322 11941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:13.784493 11941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:13.784745 11941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:13.785080 11941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:13.785136 11941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.785177 11941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:13.785207 11941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.791311 11941 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:40607
I20260928 12:58:13.791355 12069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:40607 every 8 connection(s)
I20260928 12:58:13.791736 11941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:13.801286 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:13.792609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:58965 (local address 127.11.11.126:43177)
0928 12:58:13.792670 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:13.792672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:13.792881 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:58:13.793881 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:13.793882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:13.793896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:13.793949 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:13.794575 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.795133 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.796271 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.796486 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.796543 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:13.798704 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:13.798705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:13.798706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:13.798712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:13.799964 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.800410 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.800410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.800411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.800464 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.800690 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.800690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.800691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.800796 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:13.800857 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:13.801109 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 12:58:13.801181 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:13.801434 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 11941
I20260928 12:58:13.801507 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:13.801563 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:13.792586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43177 (local address 127.11.11.65:58965)
0928 12:58:13.792750 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:13.792788 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:13.793780 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:13.794015 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:13.794018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:13.794288 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:13.794500 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:13.794504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.795225 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.795226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:13.796191 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:13.796194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.796560 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.796561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:13.796671 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:13.797175 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:13.797184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:13.798596 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:13.800086 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:13.800100 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:13.800102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:13.800341 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:13.800526 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:13.800526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:13.800527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:13.800637 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:13.800871 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:13.800876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:13.800998 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 12:58:13.801084 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:58:13.801211 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260928 12:58:13.801680 11309 external_mini_cluster.cc:1428] Setting key 2bea089e556a0bff5ea0408dbdcc38f2
I20260928 12:58:13.802263 12070 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43177
I20260928 12:58:13.802384 12070 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:13.802618 12070 heartbeater.cc:511] Master 127.11.11.126:43177 requested a full tablet report, sending...
I20260928 12:58:13.803164 11878 ts_manager.cc:194] Registered new tserver with Master: 7d7c8d2ce1f349fb9c36b541a3b3f224 (127.11.11.65:40607)
I20260928 12:58:13.803865 11878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:58965
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:13.831918 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:13.916218 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:13.916435 12075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:13.916460 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:13.917686 12075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:13.917734 12075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:13.917762 12075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:13.919307 12075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:13.919581 12075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:13.919847 12075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:13.925151 12083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:13.925185 12081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:13.925298 12075 server_base.cc:1034] running on GCE node
W20260928 12:58:13.925179 12080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:13.925650 12075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:13.925920 12075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:13.927103 12075 hybrid_clock.cc:648] HybridClock initialized: now 1790600293927082 us; error 37 us; skew 500 ppm
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:13.929868 12075 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:13.930428 12075 webserver.cc:460] Webserver started at http://127.11.11.66:33733/ using document root <none> and password file <none>
I20260928 12:58:13.930632 12075 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:13.930677 12075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:13.930781 12075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:13.931831 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "7a16384448fd4574843ac19912e2690a"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "0368ce22b3455ff50fb195c094189724"
server_key_iv: "e5a2e7d3a15bbe15aad97b2d6b333ea2"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.932227 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "7a16384448fd4574843ac19912e2690a"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "0368ce22b3455ff50fb195c094189724"
server_key_iv: "e5a2e7d3a15bbe15aad97b2d6b333ea2"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.933593 12075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:13.934350 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.934533 12075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:13.934626 12075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "7a16384448fd4574843ac19912e2690a"
format_stamp: "Formatted at 2026-09-28 12:58:13 on dist-test-slave-8sjz"
server_key: "0368ce22b3455ff50fb195c094189724"
server_key_iv: "e5a2e7d3a15bbe15aad97b2d6b333ea2"
server_key_version: "encryptionkey@0"
I20260928 12:58:13.934695 12075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:13.953101 12075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:13.955729 12075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:13.955852 12075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:13.956130 12075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:13.956439 12075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:13.956477 12075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.956501 12075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:13.956513 12075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:13.962945 12075 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:35833
I20260928 12:58:13.963019 12203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:35833 every 8 connection(s)
I20260928 12:58:13.963354 12075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:13.967137 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12075
I20260928 12:58:13.967236 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:13.967442 11309 external_mini_cluster.cc:1428] Setting key 2942e408996f75df259bbfeabe32bd0e
Sep 28 12:58:13 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:13.974694 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:13.964214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:44121 (local address 127.11.11.126:43177)
0928 12:58:13.964283 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:13.964285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:13.964508 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:58:13.965423 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:13.965425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:13.965435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:13.965492 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:13.966206 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.966691 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.968478 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.968671 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.968728 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:13.971429 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:13.971430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:13.971431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:13.971439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:13.973207 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.973685 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.973687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.973687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.973751 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:13.974016 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:13.974017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:13.974018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:13.974142 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:13.974214 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:13.974513 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 12:58:13.974602 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:13.974974 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:13.964210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43177 (local address 127.11.11.66:44121)
0928 12:58:13.964378 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:13.964416 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:13.965328 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:13.965553 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:13.965557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:13.965841 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:13.966124 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:13.966128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.966775 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.966775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:13.968335 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:13.968339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:13.968761 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:13.968762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:13.968968 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:13.969655 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:13.969667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:13.971323 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:13.973337 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:13.973345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:13.973348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:13.973593 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:13.973821 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:13.973821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:13.973822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:13.973938 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:13.974217 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:13.974225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:13.974370 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:58:13.974478 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:58:13.974647 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260928 12:58:13.975582 12204 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43177
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 12:58:13.975703 12204 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:13.975926 12204 heartbeater.cc:511] Master 127.11.11.126:43177 requested a full tablet report, sending...
I20260928 12:58:13.976349 11878 ts_manager.cc:194] Registered new tserver with Master: 7a16384448fd4574843ac19912e2690a (127.11.11.66:35833)
I20260928 12:58:13.976759 11878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:44121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:13.995641 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:14.086565 12209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:14.086812 12209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:14.086840 12209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:14.088075 12209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:14.088124 12209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:14.088156 12209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:14.089718 12209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:14.089937 12209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:14.090132 12209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:14.095818 12214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:14.095844 12215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:14.095932 12217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:14.096163 12209 server_base.cc:1034] running on GCE node
I20260928 12:58:14.096395 12209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:14.096666 12209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:14.097862 12209 hybrid_clock.cc:648] HybridClock initialized: now 1790600294097836 us; error 41 us; skew 500 ppm
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:14.100574 12209 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:14.100986 12209 webserver.cc:460] Webserver started at http://127.11.11.67:40829/ using document root <none> and password file <none>
I20260928 12:58:14.101236 12209 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:14.101282 12209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:14.101429 12209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:14.102643 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "cea26b173d014330a032392227641f40"
format_stamp: "Formatted at 2026-09-28 12:58:14 on dist-test-slave-8sjz"
server_key: "41a979f1d4cdd9de096832535eabd5bb"
server_key_iv: "13d21ba9a2401cf11ed683d87c2711d0"
server_key_version: "encryptionkey@0"
I20260928 12:58:14.103013 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "cea26b173d014330a032392227641f40"
format_stamp: "Formatted at 2026-09-28 12:58:14 on dist-test-slave-8sjz"
server_key: "41a979f1d4cdd9de096832535eabd5bb"
server_key_iv: "13d21ba9a2401cf11ed683d87c2711d0"
server_key_version: "encryptionkey@0"
I20260928 12:58:14.104507 12209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:14.105360 12224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:14.105571 12209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:14.105647 12209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "cea26b173d014330a032392227641f40"
format_stamp: "Formatted at 2026-09-28 12:58:14 on dist-test-slave-8sjz"
server_key: "41a979f1d4cdd9de096832535eabd5bb"
server_key_iv: "13d21ba9a2401cf11ed683d87c2711d0"
server_key_version: "encryptionkey@0"
I20260928 12:58:14.105749 12209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:14.124819 12209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:14.127200 12209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:14.127324 12209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:14.127600 12209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:14.127926 12209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:14.127961 12209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:14.127985 12209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:14.128029 12209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:14.133800 12209 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:33403
I20260928 12:58:14.133888 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:33403 every 8 connection(s)
I20260928 12:58:14.134210 12209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:14.141170 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12209
I20260928 12:58:14.141281 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:14.141458 11309 external_mini_cluster.cc:1428] Setting key 6b8353dbfee7f3f4234218797481ff91
I20260928 12:58:14.144351 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.135049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:56831 (local address 127.11.11.126:43177)
0928 12:58:14.135110 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.135111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.135322 (+ 211us) server_negotiation.cc:407] Connection header received
0928 12:58:14.136315 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.136316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.136327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.136391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:14.137095 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.137662 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.138788 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.138975 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.139033 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.141387 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:14.141389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:14.141389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:14.141398 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:14.143006 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.143442 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.143443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.143444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.143504 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.143743 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.143743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.143743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.143861 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:14.143915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.144209 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.144286 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:58:14.144544 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.135010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43177 (local address 127.11.11.67:56831)
0928 12:58:14.135195 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.135237 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.136223 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.136457 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.136461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.136758 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:14.136962 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.136967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.137775 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.137776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.138696 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.138699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.139049 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.139050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.139151 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.139726 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:14.139735 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:14.141264 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:14.143096 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:14.143103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:14.143105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:14.143346 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:14.143575 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:14.143576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:14.143576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:14.143686 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:14.143906 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:14.143910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:14.144085 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.144191 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.144324 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":115,"threads_started":1}
I20260928 12:58:14.144974 12338 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43177
I20260928 12:58:14.145071 12338 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:14.145215 12338 heartbeater.cc:511] Master 127.11.11.126:43177 requested a full tablet report, sending...
I20260928 12:58:14.145581 11878 ts_manager.cc:194] Registered new tserver with Master: cea26b173d014330a032392227641f40 (127.11.11.67:33403)
I20260928 12:58:14.145913 11878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:56831
I20260928 12:58:14.153627 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:58:14.161952 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.155298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.11.11.126:43177)
0928 12:58:14.155371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.155374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.155462 (+ 88us) server_negotiation.cc:407] Connection header received
0928 12:58:14.155543 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.155544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.155561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.155626 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:14.156272 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.156760 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.157580 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.157800 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.157879 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.159134 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:14.159135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:14.159135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:14.159143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:14.160686 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.161058 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.161059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.161059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.161118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.161324 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.161325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.161326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.161485 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:14.161592 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.161753 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.161835 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:58:14.164304 11878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:14.165848 11878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:14.172755 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.11.11.65:40607)
0928 12:58:14.169046 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.169051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.169059 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:14.169094 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.169096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.169111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.169343 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:14.169423 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170477 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.171765 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172342 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172441 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172506 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.172633 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.172676 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":86,"threads_started":1}
I20260928 12:58:14.172751 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:40607 (local address 127.0.0.1:49120)
0928 12:58:14.168745 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.168754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.168796 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.169226 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.169226 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.169229 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:14.169267 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.169268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170573 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.170574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.171675 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.171677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172443 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.172522 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172535 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.172608 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.172661 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260928 12:58:14.173005 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33403 (local address 127.0.0.1:59092)
0928 12:58:14.168589 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.168629 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.168700 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.169023 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.169026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.169029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:14.169100 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.169102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170540 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.170541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.171506 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.171509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172545 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.172621 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172625 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.172675 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.172718 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260928 12:58:14.173144 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.11.11.67:33403)
0928 12:58:14.168695 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.168701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.168713 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:14.168813 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.168815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.168832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.168920 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:14.169190 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170418 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.171588 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172379 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172452 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172505 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.172796 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.172984 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260928 12:58:14.173295 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.11.11.66:35833)
0928 12:58:14.169036 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.169040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.169050 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:14.169124 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.169126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.169141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.169221 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:14.169454 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170155 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.171793 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172533 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172593 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172655 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.173163 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.173214 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260928 12:58:14.173570 12004 tablet_service.cc:1467] Processing CreateTablet for tablet 3daa92010949437498c36ce291051f06 (DEFAULT_TABLE table=test-table [id=2ce472891d2342329b51349bc04820af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:14.173807 12272 tablet_service.cc:1467] Processing CreateTablet for tablet 3daa92010949437498c36ce291051f06 (DEFAULT_TABLE table=test-table [id=2ce472891d2342329b51349bc04820af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:14.173928 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daa92010949437498c36ce291051f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:14.174134 12272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daa92010949437498c36ce291051f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:14.176529 12355 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Bootstrap starting.
I20260928 12:58:14.176856 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.168657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:35833 (local address 127.0.0.1:48518)
0928 12:58:14.168869 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.168879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.168925 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.169316 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.169317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.169319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:14.169374 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.169376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.170283 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.170284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.171648 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.171651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.172688 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.172688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.172736 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.172740 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.172785 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.172821 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":33,"thread_start_us":64,"threads_started":1}
I20260928 12:58:14.177066 12356 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: Bootstrap starting.
I20260928 12:58:14.177274 12355 tablet_bootstrap.cc:654] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:14.177574 12138 tablet_service.cc:1467] Processing CreateTablet for tablet 3daa92010949437498c36ce291051f06 (DEFAULT_TABLE table=test-table [id=2ce472891d2342329b51349bc04820af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:14.177726 12356 tablet_bootstrap.cc:654] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:14.177888 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3daa92010949437498c36ce291051f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:14.177907 12355 log.cc:824] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:14.178288 12356 log.cc:824] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:14.178706 12355 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: No bootstrap required, opened a new log
I20260928 12:58:14.178800 12355 ts_tablet_manager.cc:1397] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:14.179069 12356 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: No bootstrap required, opened a new log
I20260928 12:58:14.179136 12356 ts_tablet_manager.cc:1397] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:14.180946 12359 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Bootstrap starting.
I20260928 12:58:14.180886 12355 raft_consensus.cc:348] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.181080 12355 raft_consensus.cc:374] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:14.181138 12355 raft_consensus.cc:729] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7c8d2ce1f349fb9c36b541a3b3f224, State: Initialized, Role: FOLLOWER
I20260928 12:58:14.181119 12356 raft_consensus.cc:348] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.181281 12356 raft_consensus.cc:374] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:14.181272 12355 consensus_queue.cc:260] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.181322 12356 raft_consensus.cc:729] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea26b173d014330a032392227641f40, State: Initialized, Role: FOLLOWER
I20260928 12:58:14.181457 12356 consensus_queue.cc:260] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.181574 12355 ts_tablet_manager.cc:1428] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:14.181686 12070 heartbeater.cc:503] Master 127.11.11.126:43177 was elected leader, sending a full tablet report...
I20260928 12:58:14.181737 12356 ts_tablet_manager.cc:1428] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:14.181839 12338 heartbeater.cc:503] Master 127.11.11.126:43177 was elected leader, sending a full tablet report...
I20260928 12:58:14.182027 12359 tablet_bootstrap.cc:654] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:14.182621 12359 log.cc:824] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:14.183383 12359 tablet_bootstrap.cc:492] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: No bootstrap required, opened a new log
I20260928 12:58:14.183444 12359 ts_tablet_manager.cc:1397] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:58:14.185032 12359 raft_consensus.cc:348] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.185169 12359 raft_consensus.cc:374] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:14.185192 12359 raft_consensus.cc:729] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a16384448fd4574843ac19912e2690a, State: Initialized, Role: FOLLOWER
I20260928 12:58:14.185272 12359 consensus_queue.cc:260] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.185511 12359 ts_tablet_manager.cc:1428] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:58:14.185518 12204 heartbeater.cc:503] Master 127.11.11.126:43177 was elected leader, sending a full tablet report...
W20260928 12:58:14.213932 12205 tablet.cc:2369] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:14.232709 12360 raft_consensus.cc:482] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:14.232847 12360 raft_consensus.cc:504] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.233321 12360 leader_election.cc:288] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea26b173d014330a032392227641f40 (127.11.11.67:33403), 7a16384448fd4574843ac19912e2690a (127.11.11.66:35833)
I20260928 12:58:14.237728 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.233748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:44395 (local address 127.11.11.67:33403)
0928 12:58:14.233792 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.233794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.233817 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:58:14.233852 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.233853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.233862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.233928 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:14.234151 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.234924 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.236639 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.237258 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.237340 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.237349 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.237610 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.237657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:14.237711 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.233603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33403 (local address 127.11.11.65:44395)
0928 12:58:14.233685 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.233699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.233767 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.233960 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.233961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.233963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:14.234055 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.234057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.235076 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.235077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.236521 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.236525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.237372 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.237373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.237484 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.237491 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.237562 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.237613 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":113}
I20260928 12:58:14.237730 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.233603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:35833 (local address 127.11.11.65:57207)
0928 12:58:14.233838 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:14.233846 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:14.233900 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:14.234070 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:14.234070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:14.234072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:14.234132 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.234134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.235076 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.235077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.236521 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:14.236525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.237372 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.237373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:14.237484 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.237491 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:58:14.237562 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:58:14.237663 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":108,"thread_start_us":135,"threads_started":1}
I20260928 12:58:14.237782 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.233714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:57207 (local address 127.11.11.66:35833)
0928 12:58:14.233763 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.233766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.233916 (+ 150us) server_negotiation.cc:407] Connection header received
0928 12:58:14.233996 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.233997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.234008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.234045 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:14.234212 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.234966 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.236641 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.237254 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.237341 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.237369 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.237610 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.237660 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:14.238150 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daa92010949437498c36ce291051f06" candidate_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a16384448fd4574843ac19912e2690a" is_pre_election: true
I20260928 12:58:14.238307 12292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daa92010949437498c36ce291051f06" candidate_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea26b173d014330a032392227641f40" is_pre_election: true
I20260928 12:58:14.238370 12158 raft_consensus.cc:2454] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7c8d2ce1f349fb9c36b541a3b3f224 in term 0.
I20260928 12:58:14.238479 12292 raft_consensus.cc:2454] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7c8d2ce1f349fb9c36b541a3b3f224 in term 0.
I20260928 12:58:14.238670 11957 leader_election.cc:302] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a16384448fd4574843ac19912e2690a, 7d7c8d2ce1f349fb9c36b541a3b3f224; no voters:
I20260928 12:58:14.238829 12360 raft_consensus.cc:2781] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:14.238909 12360 raft_consensus.cc:482] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:14.238926 12360 raft_consensus.cc:3037] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:14.239635 12360 raft_consensus.cc:504] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.239862 12360 leader_election.cc:288] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [CANDIDATE]: Term 1 election: Requested vote from peers cea26b173d014330a032392227641f40 (127.11.11.67:33403), 7a16384448fd4574843ac19912e2690a (127.11.11.66:35833)
I20260928 12:58:14.240002 12292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daa92010949437498c36ce291051f06" candidate_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea26b173d014330a032392227641f40"
I20260928 12:58:14.240032 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3daa92010949437498c36ce291051f06" candidate_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a16384448fd4574843ac19912e2690a"
I20260928 12:58:14.240114 12292 raft_consensus.cc:3037] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:14.240134 12158 raft_consensus.cc:3037] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:14.241026 12158 raft_consensus.cc:2454] T 3daa92010949437498c36ce291051f06 P 7a16384448fd4574843ac19912e2690a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7c8d2ce1f349fb9c36b541a3b3f224 in term 1.
I20260928 12:58:14.241026 12292 raft_consensus.cc:2454] T 3daa92010949437498c36ce291051f06 P cea26b173d014330a032392227641f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7c8d2ce1f349fb9c36b541a3b3f224 in term 1.
I20260928 12:58:14.241231 11959 leader_election.cc:302] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7c8d2ce1f349fb9c36b541a3b3f224, cea26b173d014330a032392227641f40; no voters:
I20260928 12:58:14.241350 12360 raft_consensus.cc:2781] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:14.241534 12360 raft_consensus.cc:686] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7c8d2ce1f349fb9c36b541a3b3f224, State: Running, Role: LEADER
I20260928 12:58:14.241642 12360 consensus_queue.cc:237] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } }
I20260928 12:58:14.242570 11879 catalog_manager.cc:5495] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7c8d2ce1f349fb9c36b541a3b3f224 (127.11.11.65). New cstate: current_term: 1 leader_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7c8d2ce1f349fb9c36b541a3b3f224" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cea26b173d014330a032392227641f40" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a16384448fd4574843ac19912e2690a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 35833 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:14.269357 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.261762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.11.11.65:40607)
0928 12:58:14.261857 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.261859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.261892 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:58:14.261990 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.261991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.262005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.262083 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:14.262658 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.263152 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.263907 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.264170 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.264259 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.266521 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:14.266536 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:14.266536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:14.266543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:14.268125 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.268506 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.268510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.268510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.268576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.268783 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.268784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.268784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.268944 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:14.269107 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.269145 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.269239 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
W20260928 12:58:14.292510 12071 tablet.cc:2369] T 3daa92010949437498c36ce291051f06 P 7d7c8d2ce1f349fb9c36b541a3b3f224: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:14.293457 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.285948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.11.11.65:40607)
0928 12:58:14.286022 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.286025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.286115 (+ 90us) server_negotiation.cc:407] Connection header received
0928 12:58:14.286238 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.286238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.286247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.286306 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:14.286870 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.287481 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.288326 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.288580 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.288657 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.290816 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:14.290818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:14.290818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:14.290824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:14.292313 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.292667 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.292668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.292669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.292770 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.292957 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.292957 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.292958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.293057 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:14.293138 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.293292 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.293356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[11834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:14.317715 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:14.309607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.11.11.65:40607)
0928 12:58:14.309686 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:14.309689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:14.309839 (+ 150us) server_negotiation.cc:407] Connection header received
0928 12:58:14.309920 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:14.309921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:14.309934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:14.310013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:14.310655 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.311189 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.312023 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:14.312294 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:14.312391 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:14.315044 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:14.315046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:14.315046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:14.315055 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:14.316430 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.316836 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.316838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.316838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.316903 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:14.317124 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:14.317125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:14.317125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:14.317243 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:14.317353 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:14.317502 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 12:58:14.317597 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 12:58:14.318264 11984 tablet_service.cc:2908] Rejecting scan request for tablet 3daa92010949437498c36ce291051f06: Uninitialized: clean time has not yet been initialized
I20260928 12:58:14.319195 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11941
I20260928 12:58:14.324570 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12075
I20260928 12:58:14.328987 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12209
I20260928 12:58:14.333515 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 11847
2026-09-28T12:58:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2033 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[12379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[12379](info): set up 2 sockets
Sep 28 12:58:14 dist-test-slave-8sjz krb5kdc[12379](info): commencing operation
krb5kdc: starting...
W20260928 12:58:15.380578 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:15.503404 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:39549
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:15.586314 12392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:15.586551 12392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:15.586575 12392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:15.587755 12392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:15.587800 12392 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:58:15.587813 12392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:15.587827 12392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:15.587841 12392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:15.587852 12392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:15.589468 12392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:39549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:15.589797 12392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:15.590057 12392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:15.595397 12397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:15.595397 12398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:15.595700 12392 server_base.cc:1034] running on GCE node
W20260928 12:58:15.595775 12400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:15.596091 12392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:15.596462 12392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:15.597609 12392 hybrid_clock.cc:648] HybridClock initialized: now 1790600295597595 us; error 34 us; skew 500 ppm
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:15.605396 12392 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:15.606019 12392 webserver.cc:460] Webserver started at http://127.11.11.126:42905/ using document root <none> and password file <none>
I20260928 12:58:15.606254 12392 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:15.606302 12392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:15.606400 12392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:15.607501 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "8a939aa716de4c17a3664766f2b04c09"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "6215399e07b10c447b79c97dad4ddbf0"
server_key_iv: "f214e75fc4e7d88e3fb915b83baaaf3a"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.607823 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "8a939aa716de4c17a3664766f2b04c09"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "6215399e07b10c447b79c97dad4ddbf0"
server_key_iv: "f214e75fc4e7d88e3fb915b83baaaf3a"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.609309 12392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:58:15.610069 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.610320 12392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:15.610420 12392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "8a939aa716de4c17a3664766f2b04c09"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "6215399e07b10c447b79c97dad4ddbf0"
server_key_iv: "f214e75fc4e7d88e3fb915b83baaaf3a"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.610482 12392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:15.620532 12392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:15.622985 12392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:15.623104 12392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:15.628106 12392 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:39549
I20260928 12:58:15.628157 12469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:39549 every 8 connection(s)
I20260928 12:58:15.628523 12392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:15.629166 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:15.631563 12470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Bootstrap starting.
I20260928 12:58:15.632215 12470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:15.632525 12470 log.cc:824] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:15.633306 12470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: No bootstrap required, opened a new log
I20260928 12:58:15.635473 12470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } }
I20260928 12:58:15.635622 12470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:15.635645 12470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a939aa716de4c17a3664766f2b04c09, State: Initialized, Role: FOLLOWER
I20260928 12:58:15.635848 12470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } }
I20260928 12:58:15.635929 12470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:15.635970 12470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:15.636020 12470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:15.636617 12470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } }
I20260928 12:58:15.636704 12470 leader_election.cc:302] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a939aa716de4c17a3664766f2b04c09; no voters:
I20260928 12:58:15.636904 12470 leader_election.cc:288] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:15.636988 12473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:15.637166 12470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:15.637174 12473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [term 1 LEADER]: Becoming Leader. State: Replica: 8a939aa716de4c17a3664766f2b04c09, State: Running, Role: LEADER
I20260928 12:58:15.637312 12473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } }
I20260928 12:58:15.637712 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a939aa716de4c17a3664766f2b04c09. Latest consensus state: current_term: 1 leader_uuid: "8a939aa716de4c17a3664766f2b04c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } } }
I20260928 12:58:15.637722 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a939aa716de4c17a3664766f2b04c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a939aa716de4c17a3664766f2b04c09" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 39549 } } }
I20260928 12:58:15.637943 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:15.638140 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:15.638173 12482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:15.638693 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12392
I20260928 12:58:15.638764 12482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:15.638789 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:15.638953 11309 external_mini_cluster.cc:1428] Setting key 483f13b42d9b266e5153e3578767f1da
I20260928 12:58:15.640484 12482 catalog_manager.cc:1347] Generated new cluster ID: 62254a4e458045e4a55708c2e2a1aa89
I20260928 12:58:15.640540 12482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:15.651396 12482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:15.651571 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.639502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.11.11.126:39549)
0928 12:58:15.639783 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:15.639794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:15.639830 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:58:15.639949 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:15.639957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:15.640045 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:15.640253 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:15.641005 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.642418 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.643496 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.644276 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.644397 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.647385 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:15.647390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:15.647390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:15.647405 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:15.649472 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.649902 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.649906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.649906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.650016 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.650250 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.650251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.650252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.650615 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:15.650771 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:15.651112 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 12:58:15.651263 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":77,"threads_started":1}
I20260928 12:58:15.652115 12482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:15.655098 12482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Generated new TSK 0
I20260928 12:58:15.655278 12482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:15.687258 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:39549
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:15.771829 12496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:15.772054 12496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:15.772081 12496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:15.773242 12496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:15.773280 12496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:15.773301 12496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:15.773319 12496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:15.774686 12496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:15.774890 12496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:15.775089 12496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:15.779948 12502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:15.779958 12501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:15.779958 12504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:15.780300 12496 server_base.cc:1034] running on GCE node
I20260928 12:58:15.780562 12496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:15.780793 12496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:15.781998 12496 hybrid_clock.cc:648] HybridClock initialized: now 1790600295781986 us; error 41 us; skew 500 ppm
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:15.784745 12496 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:15.785192 12496 webserver.cc:460] Webserver started at http://127.11.11.65:39921/ using document root <none> and password file <none>
I20260928 12:58:15.785439 12496 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:15.785508 12496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:15.785655 12496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:15.786706 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "5d9edbc7c90441df9a91ff3b016e0bb3"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "a6f93cf825229d3f256ce28563179fbb"
server_key_iv: "4fc7d8f96e465ccbef9218e26f8b4be8"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.787060 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "5d9edbc7c90441df9a91ff3b016e0bb3"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "a6f93cf825229d3f256ce28563179fbb"
server_key_iv: "4fc7d8f96e465ccbef9218e26f8b4be8"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.788527 12496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:15.789245 12511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.789404 12496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:15.789505 12496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "5d9edbc7c90441df9a91ff3b016e0bb3"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "a6f93cf825229d3f256ce28563179fbb"
server_key_iv: "4fc7d8f96e465ccbef9218e26f8b4be8"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.789578 12496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:15.805891 12496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:15.807873 12496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:15.808068 12496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:15.808593 12496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:15.808658 12496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.808737 12496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:15.808769 12496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.815871 12496 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38693
I20260928 12:58:15.815950 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38693 every 8 connection(s)
I20260928 12:58:15.816285 12496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:15.819526 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.809262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:52327 (local address 127.11.11.126:39549)
0928 12:58:15.809318 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:15.809320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:15.809555 (+ 235us) server_negotiation.cc:407] Connection header received
0928 12:58:15.810121 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:15.810122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:15.810135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:15.810200 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:15.811118 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.811993 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.813733 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.814074 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.814162 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.816975 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:15.816977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:15.816977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:15.816984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:15.818354 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.818751 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.818752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.818753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.818818 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.819042 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.819042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.819043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.819148 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:15.819207 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:15.819399 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:58:15.819463 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:15.819710 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.809250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.65:52327)
0928 12:58:15.809427 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:15.809463 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:15.810018 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:15.810282 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:15.810287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:15.810690 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:15.810988 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.810996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.812134 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.812135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.813646 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.813649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.814162 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.814162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.814278 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.814991 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:15.815001 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:15.816865 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:15.818431 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.818438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.818439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.818656 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.818884 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.818885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.818885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.818986 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.819187 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:15.819190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:15.819308 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 12:58:15.819384 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:58:15.819502 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 12:58:15.820283 12627 heartbeater.cc:347] Connected to a master server at 127.11.11.126:39549
I20260928 12:58:15.820369 12627 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:15.820570 12627 heartbeater.cc:511] Master 127.11.11.126:39549 requested a full tablet report, sending...
I20260928 12:58:15.821089 12423 ts_manager.cc:194] Registered new tserver with Master: 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693)
I20260928 12:58:15.822007 12423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:52327
I20260928 12:58:15.822577 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12496
I20260928 12:58:15.822649 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:15.822803 11309 external_mini_cluster.cc:1428] Setting key 8cd316d20f08b7150f46c8af493db591
I20260928 12:58:15.827643 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.821490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:42009 (local address 127.11.11.126:39549)
0928 12:58:15.821530 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:15.821532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:15.821600 (+ 68us) server_negotiation.cc:407] Connection header received
0928 12:58:15.821660 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:15.821661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:15.821669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:15.821709 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:15.822054 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.822730 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.823667 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.823852 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.823908 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.824991 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:15.824993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:15.824993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:15.825001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:15.826451 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.826858 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.826859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.826859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.826911 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.827118 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.827119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.827119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.827273 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:15.827340 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:15.827516 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:58:15.827580 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:15.827746 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.821398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.65:42009)
0928 12:58:15.821533 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:15.821544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:15.821608 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:15.821725 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:15.821726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:15.821924 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:15.821986 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.821987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.822845 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.822846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.823576 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.823579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.823929 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.823930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.824017 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.824451 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:15.824457 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:15.824888 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:15.826540 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.826541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.826541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.826784 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.826977 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.826978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.826979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.827055 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.827347 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:15.827349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:15.827409 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 12:58:15.827492 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:58:15.827609 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:15.850584 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:39549
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:15.934059 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:15.934342 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:15.934382 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:15.935626 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:15.935716 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:15.935757 12636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:15.935815 12636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:15.937501 12636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:15.937800 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:15.938077 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:15.943276 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:15.943250 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:15.943250 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:15.943356 12636 server_base.cc:1034] running on GCE node
I20260928 12:58:15.943807 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:15.944108 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:15.945292 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790600295945277 us; error 47 us; skew 500 ppm
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:15.947844 12636 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:15.948364 12636 webserver.cc:460] Webserver started at http://127.11.11.66:42179/ using document root <none> and password file <none>
I20260928 12:58:15.948604 12636 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:15.948674 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:15.948832 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:15.949904 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "65a8c9801a3043f5b26d69481e1211fa"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "373c266445762ad284943e1b3e62b153"
server_key_iv: "56f5fffb94f71e6d4cb21e0ec4e94371"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.950284 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "65a8c9801a3043f5b26d69481e1211fa"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "373c266445762ad284943e1b3e62b153"
server_key_iv: "56f5fffb94f71e6d4cb21e0ec4e94371"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.951720 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:58:15.952512 12651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.952668 12636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.952777 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "65a8c9801a3043f5b26d69481e1211fa"
format_stamp: "Formatted at 2026-09-28 12:58:15 on dist-test-slave-8sjz"
server_key: "373c266445762ad284943e1b3e62b153"
server_key_iv: "56f5fffb94f71e6d4cb21e0ec4e94371"
server_key_version: "encryptionkey@0"
I20260928 12:58:15.952853 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:15.964870 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:15.967212 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:15.967411 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:15.968029 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:15.968144 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:15.968189 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:15.968219 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:58:15 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:15.976404 12636 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:46697
I20260928 12:58:15.976840 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:15.977269 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:46697 every 8 connection(s)
I20260928 12:58:15.979280 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.968505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:38917 (local address 127.11.11.126:39549)
0928 12:58:15.968574 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:15.968577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:15.968843 (+ 266us) server_negotiation.cc:407] Connection header received
0928 12:58:15.969464 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:15.969465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:15.969481 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:15.969571 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:15.970283 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.971130 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.972546 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.972788 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.972884 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.975713 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:15.975715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:15.975767 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:15.975793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:15.977689 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.978211 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.978213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.978213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.978278 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.978562 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.978563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.978564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.978701 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:15.978786 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:15.979082 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 12:58:15.979173 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:58:15.979501 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.968508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.66:38917)
0928 12:58:15.968691 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:15.968736 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:15.969308 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:15.969679 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:15.969683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:15.969949 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:15.970162 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.970167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.971265 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.971266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.972437 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.972440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.972895 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.972896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.973035 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.973644 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:15.973656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:15.975557 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:15.977783 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.977789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.977792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.978098 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.978350 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.978351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.978352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.978479 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.978774 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:15.978780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:15.978935 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 12:58:15.979058 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 12:58:15.979249 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260928 12:58:15.980073 12767 heartbeater.cc:347] Connected to a master server at 127.11.11.126:39549
I20260928 12:58:15.980171 12767 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:15.980381 12767 heartbeater.cc:511] Master 127.11.11.126:39549 requested a full tablet report, sending...
I20260928 12:58:15.980818 12424 ts_manager.cc:194] Registered new tserver with Master: 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697)
I20260928 12:58:15.981339 12424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:38917
I20260928 12:58:15.985671 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12636
I20260928 12:58:15.985807 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:15.986009 11309 external_mini_cluster.cc:1428] Setting key 1d160c4e6f5c00f8aebe143114489b79
I20260928 12:58:15.987216 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.981351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:41657 (local address 127.11.11.126:39549)
0928 12:58:15.981397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:15.981399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:15.981486 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:58:15.981566 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:15.981566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:15.981579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:15.981650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:15.982046 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.982498 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.983359 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.983535 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.983604 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.984513 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:15.984514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:15.984515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:15.984521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:15.986013 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.986452 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.986454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.986454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.986525 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:15.986710 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:15.986711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:15.986711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:15.986863 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:15.986921 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:15.987095 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 12:58:15.987155 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:15.987377 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:15.981270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.66:41657)
0928 12:58:15.981422 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:15.981433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:15.981502 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:15.981656 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:15.981657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:15.981889 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:15.981967 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.981970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.982585 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.982586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.983285 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:15.983287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:15.983614 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:15.983614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:15.983676 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:15.984125 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:15.984130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:15.984435 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:15.986112 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.986113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.986113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.986353 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.986592 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:15.986593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:15.986593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:15.986655 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:15.986937 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:15.986938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:15.986993 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 12:58:15.987069 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:58:15.987195 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:16.011071 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:39549
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:16.092492 12776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:16.092787 12776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:16.092830 12776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:16.094079 12776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:16.094158 12776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:16.094190 12776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:16.094228 12776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:16.095793 12776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:16.096100 12776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:16.096341 12776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:16.101361 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:16.101372 12782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:16.101382 12784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:16.101766 12776 server_base.cc:1034] running on GCE node
I20260928 12:58:16.101969 12776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:16.102211 12776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:16.103392 12776 hybrid_clock.cc:648] HybridClock initialized: now 1790600296103378 us; error 29 us; skew 500 ppm
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:16.106950 12776 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:16.107486 12776 webserver.cc:460] Webserver started at http://127.11.11.67:38923/ using document root <none> and password file <none>
I20260928 12:58:16.107753 12776 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:16.107828 12776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:16.107986 12776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:16.109288 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "95b8aa3082fa4d5d905364b812035fd4"
format_stamp: "Formatted at 2026-09-28 12:58:16 on dist-test-slave-8sjz"
server_key: "88d137dd289e9b09a541b78b0ac0b24d"
server_key_iv: "dadca2f6b9a75b202295327d3e0a7f7b"
server_key_version: "encryptionkey@0"
I20260928 12:58:16.109730 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "95b8aa3082fa4d5d905364b812035fd4"
format_stamp: "Formatted at 2026-09-28 12:58:16 on dist-test-slave-8sjz"
server_key: "88d137dd289e9b09a541b78b0ac0b24d"
server_key_iv: "dadca2f6b9a75b202295327d3e0a7f7b"
server_key_version: "encryptionkey@0"
I20260928 12:58:16.111537 12776 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 12:58:16.112586 12791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:16.112874 12776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:16.112998 12776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "95b8aa3082fa4d5d905364b812035fd4"
format_stamp: "Formatted at 2026-09-28 12:58:16 on dist-test-slave-8sjz"
server_key: "88d137dd289e9b09a541b78b0ac0b24d"
server_key_iv: "dadca2f6b9a75b202295327d3e0a7f7b"
server_key_version: "encryptionkey@0"
I20260928 12:58:16.113101 12776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:16.121538 12776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:16.123425 12776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:16.123586 12776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:16.124244 12776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:16.124308 12776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:16.124351 12776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:16.124379 12776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:16.131659 12776 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40371
I20260928 12:58:16.132076 12906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40371 every 8 connection(s)
I20260928 12:58:16.132082 12776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:16.135664 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.124755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:36599 (local address 127.11.11.126:39549)
0928 12:58:16.124814 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.124816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.125110 (+ 294us) server_negotiation.cc:407] Connection header received
0928 12:58:16.126094 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.126096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.126117 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.126143 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.127071 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.127882 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.129895 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.130216 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.130297 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.133032 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.133034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.133035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.133042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.134267 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.134691 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.134692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.134693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.134758 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.135012 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.135013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.135013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.135135 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.135212 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.135484 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.135574 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:58:16.135843 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.124737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.67:36599)
0928 12:58:16.124961 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.124997 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.125897 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.126300 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.126303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.126643 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:16.126972 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.126978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.127995 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.127996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.129783 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.129786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.130328 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.130329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.130475 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.131143 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:16.131153 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:16.132933 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:16.134365 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.134372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.134374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.134592 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.134825 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.134826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.134826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.134938 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.135202 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:16.135208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:16.135351 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.135463 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.135615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260928 12:58:16.136276 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 12776
I20260928 12:58:16.136374 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:16.136556 11309 external_mini_cluster.cc:1428] Setting key a2fb1df702b4b1238f6b9da120ea9867
I20260928 12:58:16.136781 12907 heartbeater.cc:347] Connected to a master server at 127.11.11.126:39549
I20260928 12:58:16.136883 12907 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:16.137127 12907 heartbeater.cc:511] Master 127.11.11.126:39549 requested a full tablet report, sending...
I20260928 12:58:16.137521 12423 ts_manager.cc:194] Registered new tserver with Master: 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371)
I20260928 12:58:16.137997 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:58:16.138077 12423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:36599
I20260928 12:58:16.144307 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.138528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:60361 (local address 127.11.11.126:39549)
0928 12:58:16.138571 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.138573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.138685 (+ 112us) server_negotiation.cc:407] Connection header received
0928 12:58:16.138756 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.138757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.138765 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.138807 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.139192 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.139746 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.140554 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.140768 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.140851 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.141816 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.141818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.141818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.141824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.143163 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.143517 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.143519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.143519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.143572 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.143791 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.143792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.143792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.143904 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.143977 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.144138 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.144215 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:58:16.144315 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.138444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.11.11.67:60361)
0928 12:58:16.138622 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.138634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.138700 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.138819 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.138820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.139040 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:16.139110 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.139112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.139852 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.139853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.140471 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.140473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.140853 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.140854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.140926 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.141412 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:16.141417 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:16.141711 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:16.143261 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.143262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.143263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.143431 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.143656 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.143657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.143658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.143717 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.143979 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:16.143980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:16.144023 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.144116 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.144216 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:16.161777 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.154406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.11.11.126:39549)
0928 12:58:16.154478 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.154480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.154489 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:16.154515 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.154516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.154533 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.154587 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.155285 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.155893 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.156700 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.156878 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.156934 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.159141 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.159142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.159142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.159149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.160613 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.161135 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.161136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.161137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.161193 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.161392 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.161392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.161393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.161492 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.161568 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.161652 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.161713 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:58:16.163560 12423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:16.165295 12423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:16.172140 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.167924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.11.11.67:40371)
0928 12:58:16.168260 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.168266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.168276 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:16.168318 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.168321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.168337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.168420 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.168632 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.169402 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.170927 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.171651 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171730 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.171791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.171965 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.172007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":73,"threads_started":1}
I20260928 12:58:16.172147 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.167770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38693 (local address 127.0.0.1:50664)
0928 12:58:16.168089 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.168098 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.168138 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.168712 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.168712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.168714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.168751 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.168752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.169583 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.169584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.170784 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.170786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.171828 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.171894 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.171898 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.171969 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.172009 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260928 12:58:16.172405 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.167884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.11.11.65:38693)
0928 12:58:16.168554 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.168559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.168569 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:16.168606 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.168608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.168625 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.168714 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.168812 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.169487 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.170878 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.171651 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171730 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.171791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.172282 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.172325 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":83,"threads_started":1}
I20260928 12:58:16.172170 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.167719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40371 (local address 127.0.0.1:46956)
0928 12:58:16.167862 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.167908 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.167970 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.168434 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.168438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.168440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.168512 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.168514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.169493 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.169494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.170852 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.170855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.171752 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.171846 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.171863 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.171934 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.171987 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260928 12:58:16.173171 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.167957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:46697 (local address 127.0.0.1:53338)
0928 12:58:16.168196 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.168202 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.168226 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.169688 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.169689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.169690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.169795 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.169797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.171014 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.171731 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.171734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.172919 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.172920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.173011 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.173016 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.173069 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.173107 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":59,"thread_start_us":62,"threads_started":1}
I20260928 12:58:16.173123 12561 tablet_service.cc:1467] Processing CreateTablet for tablet d0cfd52c5fb84416860e9f7296fd662a (DEFAULT_TABLE table=test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.173480 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.168192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.11.11.66:46697)
0928 12:58:16.169521 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.169526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.169537 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:16.169579 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.169581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.169598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.169679 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.169873 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.170948 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.171837 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.172758 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.172828 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.172887 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.173334 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.173378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260928 12:58:16.173502 12561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cfd52c5fb84416860e9f7296fd662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.173431 12841 tablet_service.cc:1467] Processing CreateTablet for tablet d0cfd52c5fb84416860e9f7296fd662a (DEFAULT_TABLE table=test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.173738 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cfd52c5fb84416860e9f7296fd662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.176138 12930 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Bootstrap starting.
I20260928 12:58:16.176854 12701 tablet_service.cc:1467] Processing CreateTablet for tablet d0cfd52c5fb84416860e9f7296fd662a (DEFAULT_TABLE table=test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.177138 12930 tablet_bootstrap.cc:654] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.177179 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cfd52c5fb84416860e9f7296fd662a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.177453 12930 log.cc:824] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:16.178113 12930 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: No bootstrap required, opened a new log
I20260928 12:58:16.178166 12930 ts_tablet_manager.cc:1397] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:16.179836 12930 raft_consensus.cc:348] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.179965 12930 raft_consensus.cc:374] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.179986 12930 raft_consensus.cc:729] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b8aa3082fa4d5d905364b812035fd4, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.180127 12930 consensus_queue.cc:260] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.180406 12930 ts_tablet_manager.cc:1428] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:16.180480 12931 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Bootstrap starting.
I20260928 12:58:16.180507 12907 heartbeater.cc:503] Master 127.11.11.126:39549 was elected leader, sending a full tablet report...
I20260928 12:58:16.180601 12933 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Bootstrap starting.
I20260928 12:58:16.181159 12931 tablet_bootstrap.cc:654] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.181303 12933 tablet_bootstrap.cc:654] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.181516 12931 log.cc:824] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:16.181617 12933 log.cc:824] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:16.182271 12931 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: No bootstrap required, opened a new log
I20260928 12:58:16.182297 12933 tablet_bootstrap.cc:492] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: No bootstrap required, opened a new log
I20260928 12:58:16.182353 12931 ts_tablet_manager.cc:1397] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 12:58:16.182344 12933 ts_tablet_manager.cc:1397] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:16.183898 12933 raft_consensus.cc:348] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.183931 12931 raft_consensus.cc:348] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.184026 12933 raft_consensus.cc:374] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.184119 12931 raft_consensus.cc:374] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.184137 12933 raft_consensus.cc:729] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a8c9801a3043f5b26d69481e1211fa, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.184167 12931 raft_consensus.cc:729] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9edbc7c90441df9a91ff3b016e0bb3, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.184250 12933 consensus_queue.cc:260] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.184297 12931 consensus_queue.cc:260] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.184543 12767 heartbeater.cc:503] Master 127.11.11.126:39549 was elected leader, sending a full tablet report...
I20260928 12:58:16.184584 12931 ts_tablet_manager.cc:1428] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:58:16.184619 12627 heartbeater.cc:503] Master 127.11.11.126:39549 was elected leader, sending a full tablet report...
I20260928 12:58:16.184538 12933 ts_tablet_manager.cc:1428] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:58:16.221675 12934 raft_consensus.cc:482] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:16.221807 12934 raft_consensus.cc:504] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.222232 12934 leader_election.cc:288] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693), 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697)
I20260928 12:58:16.226344 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.222713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:60405 (local address 127.11.11.66:46697)
0928 12:58:16.222807 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.222809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.222817 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:16.222849 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.222850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.222857 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.222914 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.223050 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.223752 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.225347 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.225917 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.226000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.226010 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.226227 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.226268 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 12:58:16.226344 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.222551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:44509 (local address 127.11.11.65:38693)
0928 12:58:16.222650 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.222652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.222662 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:16.222693 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.222694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.222704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.222769 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.222937 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.223739 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.225347 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.225917 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.226000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.226010 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.226240 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.226275 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 12:58:16.226375 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.222436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38693 (local address 127.11.11.67:44509)
0928 12:58:16.222501 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.222515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.222598 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.222782 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.222783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.222784 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.222861 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.222863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.223852 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.223853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.225242 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.225246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.226026 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.226027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.226135 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.226142 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.226209 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.226260 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":90}
I20260928 12:58:16.226382 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.222550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:46697 (local address 127.11.11.67:60405)
0928 12:58:16.222736 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.222746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.222793 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.222932 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.222933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.222934 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.222989 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.222991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.223852 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.223853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.225242 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.225246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.226026 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.226027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.226135 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.226142 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.226209 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.226266 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":103,"thread_start_us":132,"threads_started":1}
I20260928 12:58:16.226846 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cfd52c5fb84416860e9f7296fd662a" candidate_uuid: "95b8aa3082fa4d5d905364b812035fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" is_pre_election: true
I20260928 12:58:16.226846 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cfd52c5fb84416860e9f7296fd662a" candidate_uuid: "95b8aa3082fa4d5d905364b812035fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a8c9801a3043f5b26d69481e1211fa" is_pre_election: true
I20260928 12:58:16.227017 12721 raft_consensus.cc:2454] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b8aa3082fa4d5d905364b812035fd4 in term 0.
I20260928 12:58:16.227017 12581 raft_consensus.cc:2454] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b8aa3082fa4d5d905364b812035fd4 in term 0.
I20260928 12:58:16.227293 12795 leader_election.cc:302] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edbc7c90441df9a91ff3b016e0bb3, 95b8aa3082fa4d5d905364b812035fd4; no voters:
W20260928 12:58:16.227391 12768 tablet.cc:2369] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:16.227432 12934 raft_consensus.cc:2781] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:16.227487 12934 raft_consensus.cc:482] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:16.227517 12934 raft_consensus.cc:3037] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.228328 12934 raft_consensus.cc:504] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.228469 12934 leader_election.cc:288] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693), 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697)
I20260928 12:58:16.228708 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cfd52c5fb84416860e9f7296fd662a" candidate_uuid: "95b8aa3082fa4d5d905364b812035fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3"
I20260928 12:58:16.228715 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cfd52c5fb84416860e9f7296fd662a" candidate_uuid: "95b8aa3082fa4d5d905364b812035fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a8c9801a3043f5b26d69481e1211fa"
I20260928 12:58:16.228808 12721 raft_consensus.cc:3037] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.228814 12581 raft_consensus.cc:3037] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.229550 12721 raft_consensus.cc:2454] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b8aa3082fa4d5d905364b812035fd4 in term 1.
I20260928 12:58:16.229633 12581 raft_consensus.cc:2454] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b8aa3082fa4d5d905364b812035fd4 in term 1.
I20260928 12:58:16.229766 12795 leader_election.cc:302] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a8c9801a3043f5b26d69481e1211fa, 95b8aa3082fa4d5d905364b812035fd4; no voters:
I20260928 12:58:16.229882 12934 raft_consensus.cc:2781] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:16.230044 12934 raft_consensus.cc:686] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 95b8aa3082fa4d5d905364b812035fd4, State: Running, Role: LEADER
I20260928 12:58:16.230118 12934 consensus_queue.cc:237] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } }
I20260928 12:58:16.230988 12423 catalog_manager.cc:5495] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "95b8aa3082fa4d5d905364b812035fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:16.249228 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.246800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.11.11.67:40371)
0928 12:58:16.246876 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.246878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.246886 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:16.246952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.246953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.246970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.247032 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:16.247157 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.247644 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.248492 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.248691 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.248746 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.248931 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.249024 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.249081 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.249127 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.249146 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:58:16.251324 12721 raft_consensus.cc:1261] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Refusing update from remote peer 95b8aa3082fa4d5d905364b812035fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.251330 12581 raft_consensus.cc:1261] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Refusing update from remote peer 95b8aa3082fa4d5d905364b812035fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.251758 12940 consensus_queue.cc:1035] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.251888 12934 consensus_queue.cc:1035] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.258504 12423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:16.258630 12423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:16.259452 12423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Sending DeleteTablet for 3 replicas of tablet d0cfd52c5fb84416860e9f7296fd662a
I20260928 12:58:16.259874 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cfd52c5fb84416860e9f7296fd662a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:46956
I20260928 12:58:16.259905 12561 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cfd52c5fb84416860e9f7296fd662a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260928 12:58:16.259961 12701 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cfd52c5fb84416860e9f7296fd662a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53338
I20260928 12:58:16.260336 12955 tablet_replica.cc:330] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: stopping tablet replica
I20260928 12:58:16.260450 12955 raft_consensus.cc:2229] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:16.260483 12423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:16.260665 12423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:16.260878 12955 raft_consensus.cc:2258] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.261104 12954 tablet_replica.cc:330] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: stopping tablet replica
I20260928 12:58:16.261197 12954 raft_consensus.cc:2229] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:16.261291 12954 raft_consensus.cc:2258] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.261493 12955 ts_tablet_manager.cc:1905] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.261739 12954 ts_tablet_manager.cc:1905] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.262511 12701 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6cbcc079248b08733cfb9c01e847f (DEFAULT_TABLE table=test-table [id=d57a85c8b0304d068f440a218ce60bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.262696 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6cbcc079248b08733cfb9c01e847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.264199 12955 ts_tablet_manager.cc:1918] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:16.264276 12955 log.cc:1196] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/d0cfd52c5fb84416860e9f7296fd662a
I20260928 12:58:16.264571 12955 ts_tablet_manager.cc:1939] T d0cfd52c5fb84416860e9f7296fd662a P 65a8c9801a3043f5b26d69481e1211fa: Deleting consensus metadata
I20260928 12:58:16.264609 12933 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Bootstrap starting.
I20260928 12:58:16.264607 12954 ts_tablet_manager.cc:1918] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:16.264693 12954 log.cc:1196] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/d0cfd52c5fb84416860e9f7296fd662a
I20260928 12:58:16.265197 12954 ts_tablet_manager.cc:1939] T d0cfd52c5fb84416860e9f7296fd662a P 95b8aa3082fa4d5d905364b812035fd4: Deleting consensus metadata
I20260928 12:58:16.265710 12933 tablet_bootstrap.cc:654] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.265750 12953 tablet_replica.cc:330] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: stopping tablet replica
I20260928 12:58:16.265928 12953 raft_consensus.cc:2229] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:16.266130 12953 raft_consensus.cc:2258] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.266686 12410 catalog_manager.cc:4841] TS 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371): tablet d0cfd52c5fb84416860e9f7296fd662a (table test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]) successfully deleted
I20260928 12:58:16.266826 12953 ts_tablet_manager.cc:1905] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.266858 12411 catalog_manager.cc:4841] TS 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697): tablet d0cfd52c5fb84416860e9f7296fd662a (table test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]) successfully deleted
I20260928 12:58:16.267063 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6cbcc079248b08733cfb9c01e847f (DEFAULT_TABLE table=test-table [id=d57a85c8b0304d068f440a218ce60bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.267195 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6cbcc079248b08733cfb9c01e847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.267254 12561 tablet_service.cc:1467] Processing CreateTablet for tablet 22f6cbcc079248b08733cfb9c01e847f (DEFAULT_TABLE table=test-table [id=d57a85c8b0304d068f440a218ce60bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:16.267355 12561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f6cbcc079248b08733cfb9c01e847f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.268957 12930 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Bootstrap starting.
I20260928 12:58:16.269378 12931 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Bootstrap starting.
I20260928 12:58:16.269487 12953 ts_tablet_manager.cc:1918] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:16.269578 12953 log.cc:1196] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/d0cfd52c5fb84416860e9f7296fd662a
I20260928 12:58:16.269941 12953 ts_tablet_manager.cc:1939] T d0cfd52c5fb84416860e9f7296fd662a P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting consensus metadata
I20260928 12:58:16.270318 12930 tablet_bootstrap.cc:654] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.270682 12931 tablet_bootstrap.cc:654] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.271693 12411 catalog_manager.cc:4841] TS 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693): tablet d0cfd52c5fb84416860e9f7296fd662a (table test-table [id=8fa6758f1d6d47c887d67d871e8bfee3]) successfully deleted
I20260928 12:58:16.271965 12931 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: No bootstrap required, opened a new log
I20260928 12:58:16.272006 12931 ts_tablet_manager.cc:1397] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:58:16.272432 12931 raft_consensus.cc:348] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.272516 12931 raft_consensus.cc:374] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.272541 12931 raft_consensus.cc:729] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9edbc7c90441df9a91ff3b016e0bb3, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.272583 12931 consensus_queue.cc:260] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.272687 12931 ts_tablet_manager.cc:1428] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:16.272826 12930 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: No bootstrap required, opened a new log
I20260928 12:58:16.272877 12930 ts_tablet_manager.cc:1397] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:58:16.273124 12930 raft_consensus.cc:348] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.273252 12930 raft_consensus.cc:374] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.273276 12930 raft_consensus.cc:729] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b8aa3082fa4d5d905364b812035fd4, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.273317 12933 tablet_bootstrap.cc:492] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: No bootstrap required, opened a new log
I20260928 12:58:16.273314 12930 consensus_queue.cc:260] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.273373 12933 ts_tablet_manager.cc:1397] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260928 12:58:16.273419 12930 ts_tablet_manager.cc:1428] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:16.273793 12933 raft_consensus.cc:348] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.273871 12933 raft_consensus.cc:374] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.273914 12933 raft_consensus.cc:729] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a8c9801a3043f5b26d69481e1211fa, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.273968 12933 consensus_queue.cc:260] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.274065 12933 ts_tablet_manager.cc:1428] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260928 12:58:16.317142 12628 tablet.cc:2369] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:58:16.382763 12908 tablet.cc:2369] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:16.383198 12937 raft_consensus.cc:482] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:16.383309 12937 raft_consensus.cc:504] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.383731 12937 leader_election.cc:288] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693), 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371)
I20260928 12:58:16.387662 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.383999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:38711 (local address 127.11.11.65:38693)
0928 12:58:16.384067 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.384070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.384082 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:16.384189 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.384190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.384202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.384260 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.384506 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.385233 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.386833 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.387242 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.387340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.387350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.387537 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.387588 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:58:16.387681 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.383936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38693 (local address 127.11.11.66:38711)
0928 12:58:16.383999 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.384011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.384089 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.384313 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.384314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.384315 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.384417 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.384420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.385340 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.385341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.386697 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.386701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.387338 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.387339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.387449 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.387454 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.387519 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.387577 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":22}
I20260928 12:58:16.387697 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.384148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:54053 (local address 127.11.11.67:40371)
0928 12:58:16.384210 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.384212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.384222 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:16.384280 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.384281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.384291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.384341 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.384543 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.385102 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.386797 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.387291 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.387368 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.387377 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.387585 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.387633 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:58:16.388025 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6cbcc079248b08733cfb9c01e847f" candidate_uuid: "65a8c9801a3043f5b26d69481e1211fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" is_pre_election: true
I20260928 12:58:16.388166 12581 raft_consensus.cc:2454] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a8c9801a3043f5b26d69481e1211fa in term 0.
I20260928 12:58:16.388384 12655 leader_election.cc:302] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edbc7c90441df9a91ff3b016e0bb3, 65a8c9801a3043f5b26d69481e1211fa; no voters:
I20260928 12:58:16.388456 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.383977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40371 (local address 127.11.11.66:54053)
0928 12:58:16.384152 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.384164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.384214 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.384363 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.384364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.384365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.384435 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.384437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.385207 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.385208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.386697 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.386701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.387418 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.387419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.387487 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.387491 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.388268 (+ 777us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.388334 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":37,"thread_start_us":76,"threads_started":1}
I20260928 12:58:16.388557 12937 raft_consensus.cc:2781] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:16.388621 12937 raft_consensus.cc:482] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:16.388665 12937 raft_consensus.cc:3037] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.388813 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6cbcc079248b08733cfb9c01e847f" candidate_uuid: "65a8c9801a3043f5b26d69481e1211fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b8aa3082fa4d5d905364b812035fd4" is_pre_election: true
I20260928 12:58:16.388947 12860 raft_consensus.cc:2454] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a8c9801a3043f5b26d69481e1211fa in term 0.
I20260928 12:58:16.389567 12937 raft_consensus.cc:504] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.389765 12937 leader_election.cc:288] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [CANDIDATE]: Term 1 election: Requested vote from peers 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693), 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371)
I20260928 12:58:16.389937 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6cbcc079248b08733cfb9c01e847f" candidate_uuid: "65a8c9801a3043f5b26d69481e1211fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3"
I20260928 12:58:16.390024 12581 raft_consensus.cc:3037] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.389999 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f6cbcc079248b08733cfb9c01e847f" candidate_uuid: "65a8c9801a3043f5b26d69481e1211fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b8aa3082fa4d5d905364b812035fd4"
I20260928 12:58:16.390072 12860 raft_consensus.cc:3037] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.390688 12860 raft_consensus.cc:2454] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a8c9801a3043f5b26d69481e1211fa in term 1.
I20260928 12:58:16.390697 12581 raft_consensus.cc:2454] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a8c9801a3043f5b26d69481e1211fa in term 1.
I20260928 12:58:16.390897 12655 leader_election.cc:302] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edbc7c90441df9a91ff3b016e0bb3, 65a8c9801a3043f5b26d69481e1211fa; no voters:
I20260928 12:58:16.391001 12937 raft_consensus.cc:2781] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:16.391186 12937 raft_consensus.cc:686] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 1 LEADER]: Becoming Leader. State: Replica: 65a8c9801a3043f5b26d69481e1211fa, State: Running, Role: LEADER
I20260928 12:58:16.391279 12937 consensus_queue.cc:237] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.391994 12423 catalog_manager.cc:5495] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66). New cstate: current_term: 1 leader_uuid: "65a8c9801a3043f5b26d69481e1211fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:16.456184 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.449008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.11.11.126:39549)
0928 12:58:16.449088 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.449090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.449099 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:16.449122 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.449123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.449132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.449199 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.449659 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.450151 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.450829 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.451002 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.451055 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.453497 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.453498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.453499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.453506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.454901 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.455312 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.455314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.455314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.455371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.455598 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.455599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.455600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.455712 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.455789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.455963 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.456072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:58:16.456260 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.448814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:39549 (local address 127.0.0.1:54874)
0928 12:58:16.449008 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.449017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.449067 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.449223 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.449224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.449531 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:16.449578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.449580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.450230 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.450231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.450754 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.450755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.451093 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.451094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.451160 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.451711 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:16.451716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:16.453354 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:16.454990 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.454997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.455000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.455235 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.455439 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:16.455440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:16.455440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:16.455504 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:16.455792 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:16.455800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:16.455869 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.455942 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.456129 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":86,"thread_start_us":76,"threads_started":1}
I20260928 12:58:16.458303 12423 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:54874:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:58:16.458561 12423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:16.460129 12701 tablet_service.cc:1467] Processing CreateTablet for tablet ab7f049d63514aa6bcd4702cebb340a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d69138afee2647ceb31bbc5105103a44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:16.460134 12841 tablet_service.cc:1467] Processing CreateTablet for tablet ab7f049d63514aa6bcd4702cebb340a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d69138afee2647ceb31bbc5105103a44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:16.460155 12561 tablet_service.cc:1467] Processing CreateTablet for tablet ab7f049d63514aa6bcd4702cebb340a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d69138afee2647ceb31bbc5105103a44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:16.460297 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7f049d63514aa6bcd4702cebb340a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.460299 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7f049d63514aa6bcd4702cebb340a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.460327 12561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7f049d63514aa6bcd4702cebb340a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:16.461683 12931 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: Bootstrap starting.
I20260928 12:58:16.461767 12933 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: Bootstrap starting.
I20260928 12:58:16.461778 12930 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: Bootstrap starting.
I20260928 12:58:16.462162 12930 tablet_bootstrap.cc:654] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.462195 12931 tablet_bootstrap.cc:654] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.462222 12933 tablet_bootstrap.cc:654] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:16.462828 12931 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: No bootstrap required, opened a new log
I20260928 12:58:16.462889 12931 ts_tablet_manager.cc:1397] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:16.462931 12930 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: No bootstrap required, opened a new log
I20260928 12:58:16.462944 12933 tablet_bootstrap.cc:492] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: No bootstrap required, opened a new log
I20260928 12:58:16.462985 12930 ts_tablet_manager.cc:1397] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:16.462985 12933 ts_tablet_manager.cc:1397] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:16.463063 12931 raft_consensus.cc:348] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463107 12930 raft_consensus.cc:348] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463133 12933 raft_consensus.cc:348] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463178 12930 raft_consensus.cc:374] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.463179 12931 raft_consensus.cc:374] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.463194 12933 raft_consensus.cc:374] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:16.463201 12930 raft_consensus.cc:729] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b8aa3082fa4d5d905364b812035fd4, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.463209 12931 raft_consensus.cc:729] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9edbc7c90441df9a91ff3b016e0bb3, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.463225 12933 raft_consensus.cc:729] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a8c9801a3043f5b26d69481e1211fa, State: Initialized, Role: FOLLOWER
I20260928 12:58:16.463261 12930 consensus_queue.cc:260] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463269 12933 consensus_queue.cc:260] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463263 12931 consensus_queue.cc:260] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.463354 12930 ts_tablet_manager.cc:1428] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:16.463354 12933 ts_tablet_manager.cc:1428] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:16.463362 12931 ts_tablet_manager.cc:1428] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:16.463470 12948 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.463511 12937 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.463546 12948 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.463573 12937 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.463614 12938 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.463732 12938 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.572103 12938 raft_consensus.cc:482] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:16.572229 12938 raft_consensus.cc:504] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.572654 12938 leader_election.cc:288] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697), 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371)
I20260928 12:58:16.576953 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.573061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45687 (local address 127.11.11.66:46697)
0928 12:58:16.573111 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.573117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.573183 (+ 66us) server_negotiation.cc:407] Connection header received
0928 12:58:16.573260 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.573261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.573269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.573324 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.573536 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.574304 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.575963 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.576540 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.576628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.576637 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.576845 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.576893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:16.576963 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.572889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40371 (local address 127.11.11.65:36483)
0928 12:58:16.573038 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.573051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.573150 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.573349 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.573350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.573352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.573457 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.573458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.574402 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.574403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.575837 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.575841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.576667 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.576668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.576742 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.576748 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.576814 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.576863 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":127,"thread_start_us":88,"threads_started":1}
I20260928 12:58:16.576952 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.572977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:36483 (local address 127.11.11.67:40371)
0928 12:58:16.573038 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.573041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.573167 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:58:16.573239 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.573241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.573252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.573313 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:16.573536 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.574305 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.575963 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.576566 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.576629 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.576637 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.576821 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.576871 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:58:16.577190 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.572909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:46697 (local address 127.11.11.65:45687)
0928 12:58:16.573127 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.573136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.573192 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.573349 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.573350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.573353 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:16.573452 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.573454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.574401 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.574402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.575837 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.575841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.576655 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.576657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.576742 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.576748 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.577047 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.577099 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":94,"thread_start_us":58,"threads_started":1}
I20260928 12:58:16.577466 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7f049d63514aa6bcd4702cebb340a1" candidate_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a8c9801a3043f5b26d69481e1211fa" is_pre_election: true
I20260928 12:58:16.577494 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7f049d63514aa6bcd4702cebb340a1" candidate_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b8aa3082fa4d5d905364b812035fd4" is_pre_election: true
I20260928 12:58:16.577580 12721 raft_consensus.cc:2454] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9edbc7c90441df9a91ff3b016e0bb3 in term 0.
I20260928 12:58:16.577579 12860 raft_consensus.cc:2454] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9edbc7c90441df9a91ff3b016e0bb3 in term 0.
I20260928 12:58:16.577817 12514 leader_election.cc:302] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d9edbc7c90441df9a91ff3b016e0bb3, 65a8c9801a3043f5b26d69481e1211fa, 95b8aa3082fa4d5d905364b812035fd4; no voters:
I20260928 12:58:16.577973 12938 raft_consensus.cc:2781] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:16.578020 12938 raft_consensus.cc:482] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:16.578049 12938 raft_consensus.cc:3037] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.578730 12938 raft_consensus.cc:504] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.578868 12938 leader_election.cc:288] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697), 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371)
I20260928 12:58:16.579056 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7f049d63514aa6bcd4702cebb340a1" candidate_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b8aa3082fa4d5d905364b812035fd4"
I20260928 12:58:16.579130 12860 raft_consensus.cc:3037] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.579078 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7f049d63514aa6bcd4702cebb340a1" candidate_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a8c9801a3043f5b26d69481e1211fa"
I20260928 12:58:16.579185 12721 raft_consensus.cc:3037] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:16.579751 12860 raft_consensus.cc:2454] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9edbc7c90441df9a91ff3b016e0bb3 in term 1.
I20260928 12:58:16.579900 12721 raft_consensus.cc:2454] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9edbc7c90441df9a91ff3b016e0bb3 in term 1.
I20260928 12:58:16.579926 12514 leader_election.cc:302] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edbc7c90441df9a91ff3b016e0bb3, 95b8aa3082fa4d5d905364b812035fd4; no voters:
I20260928 12:58:16.580099 12938 raft_consensus.cc:2781] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:16.580286 12938 raft_consensus.cc:686] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9edbc7c90441df9a91ff3b016e0bb3, State: Running, Role: LEADER
I20260928 12:58:16.580384 12938 consensus_queue.cc:237] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } }
I20260928 12:58:16.580793 12973 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: TxnStatusTablet state changed. Reason: New leader 5d9edbc7c90441df9a91ff3b016e0bb3. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.580888 12973 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:16.581064 12975 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:58:16.581208 12975 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:58:16.581260 12424 catalog_manager.cc:5495] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65). New cstate: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:16.610733 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.608164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38693 (local address 127.0.0.1:50674)
0928 12:58:16.608245 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.608258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.608309 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.608477 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.608477 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.608478 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:16.608535 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.608538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.609192 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.609193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.609777 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.609778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.610162 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.610163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.610239 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.610253 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.610543 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.610553 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.610609 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.610657 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:58:16.610718 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.608258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.11.11.65:38693)
0928 12:58:16.608307 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.608310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.608320 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:16.608387 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.608389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.608409 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.608464 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:16.608605 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.609100 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.609869 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.610083 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.610161 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.610361 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.610472 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.610557 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.610614 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.610634 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:16.611918 12860 raft_consensus.cc:1261] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Refusing update from remote peer 5d9edbc7c90441df9a91ff3b016e0bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.612166 12721 raft_consensus.cc:1261] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Refusing update from remote peer 5d9edbc7c90441df9a91ff3b016e0bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.612325 12973 consensus_queue.cc:1035] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.612448 12938 consensus_queue.cc:1035] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.613183 12948 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: TxnStatusTablet state changed. Reason: New leader 5d9edbc7c90441df9a91ff3b016e0bb3. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.613271 12948 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.613368 12937 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: TxnStatusTablet state changed. Reason: New leader 5d9edbc7c90441df9a91ff3b016e0bb3. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.613461 12937 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.613915 12945 mvcc.cc:205] Tried to move back new op lower bound from 7334298814920146944 to 7334298814793969664. Current Snapshot: MvccSnapshot[applied={T|T < 7334298814920146944}]
I20260928 12:58:16.613982 12938 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.614054 12938 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:16.614044 12944 mvcc.cc:205] Tried to move back new op lower bound from 7334298814920146944 to 7334298814793969664. Current Snapshot: MvccSnapshot[applied={T|T < 7334298814920146944 or (T in {7334298814920146944})}]
I20260928 12:58:16.614203 12937 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.614270 12937 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 65a8c9801a3043f5b26d69481e1211fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.614917 12973 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.615005 12973 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 5d9edbc7c90441df9a91ff3b016e0bb3: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:16.615556 12943 mvcc.cc:205] Tried to move back new op lower bound from 7334298814920146944 to 7334298814793969664. Current Snapshot: MvccSnapshot[applied={T|T < 7334298814920146944}]
I20260928 12:58:16.615654 12948 tablet_replica.cc:439] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 } } peers { permanent_uuid: "65a8c9801a3043f5b26d69481e1211fa" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 46697 } } peers { permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 } } }
I20260928 12:58:16.615723 12948 tablet_replica.cc:442] T ab7f049d63514aa6bcd4702cebb340a1 P 95b8aa3082fa4d5d905364b812035fd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:16.618804 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.616009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.11.11.66:46697)
0928 12:58:16.616077 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.616079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.616086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:16.616121 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.616121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.616140 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.616204 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:16.616406 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.617236 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.617955 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.618279 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.618349 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.618489 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.618613 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.618684 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.618712 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.618731 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:16.623766 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.621425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:57021 (local address 127.11.11.65:38693)
0928 12:58:16.621481 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.621483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.621598 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:58:16.621675 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.621676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.621686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.621735 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:16.621854 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.622302 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.623059 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.623258 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.623330 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.623505 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.623544 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.623616 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.623678 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.623696 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:58:16.623802 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.621343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38693 (local address 127.11.11.66:57021)
0928 12:58:16.621522 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.621533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.621624 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.621738 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.621739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.621740 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:16.621794 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.621795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.622389 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.622390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.622973 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.622975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.623340 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.623341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.623415 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.623428 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.623617 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.623620 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.623672 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.623712 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260928 12:58:16.626190 12860 raft_consensus.cc:1261] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Refusing update from remote peer 65a8c9801a3043f5b26d69481e1211fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.626243 12581 raft_consensus.cc:1261] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Refusing update from remote peer 65a8c9801a3043f5b26d69481e1211fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:16.626683 12960 consensus_queue.cc:1035] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9edbc7c90441df9a91ff3b016e0bb3" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.626806 12937 consensus_queue.cc:1035] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b8aa3082fa4d5d905364b812035fd4" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:16.635347 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.632686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:50573 (local address 127.11.11.66:46697)
0928 12:58:16.632755 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.632757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.632883 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:58:16.632942 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.632943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.632950 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.632993 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:16.633121 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.633615 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.634756 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.634934 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.634989 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.635168 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.635193 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.635232 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.635287 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.635302 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:16.635591 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.632612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:46697 (local address 127.11.11.65:50573)
0928 12:58:16.632825 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:16.632835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:16.632890 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:16.632997 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:16.632997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:16.632999 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:16.633060 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.633061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.633693 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.633694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.634655 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:16.634657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.635006 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.635007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:16.635076 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.635092 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:16.635232 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:16.635233 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:16.635284 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:58:16.635327 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260928 12:58:16.644713 12424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:16.644835 12424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:16.645475 12424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a939aa716de4c17a3664766f2b04c09: Sending DeleteTablet for 3 replicas of tablet 22f6cbcc079248b08733cfb9c01e847f
I20260928 12:58:16.645732 12561 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6cbcc079248b08733cfb9c01e847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:50664
I20260928 12:58:16.645831 12953 tablet_replica.cc:330] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: stopping tablet replica
I20260928 12:58:16.645843 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6cbcc079248b08733cfb9c01e847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:46956
I20260928 12:58:16.645874 12953 raft_consensus.cc:2229] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:16.645910 12953 raft_consensus.cc:2258] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.645952 12954 tablet_replica.cc:330] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: stopping tablet replica
I20260928 12:58:16.646027 12954 raft_consensus.cc:2229] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:16.646063 12954 raft_consensus.cc:2258] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.646380 12954 ts_tablet_manager.cc:1905] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.646456 12953 ts_tablet_manager.cc:1905] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.646516 12701 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f6cbcc079248b08733cfb9c01e847f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:16 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53338
I20260928 12:58:16.647560 12955 tablet_replica.cc:330] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: stopping tablet replica
I20260928 12:58:16.647605 12955 raft_consensus.cc:2229] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:16.647670 12955 raft_consensus.cc:2258] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:16.647914 12954 ts_tablet_manager.cc:1918] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:16.647982 12954 log.cc:1196] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/22f6cbcc079248b08733cfb9c01e847f
I20260928 12:58:16.648217 12954 ts_tablet_manager.cc:1939] T 22f6cbcc079248b08733cfb9c01e847f P 95b8aa3082fa4d5d905364b812035fd4: Deleting consensus metadata
I20260928 12:58:16.648262 12953 ts_tablet_manager.cc:1918] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:16.648334 12953 log.cc:1196] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/22f6cbcc079248b08733cfb9c01e847f
I20260928 12:58:16.648548 12953 ts_tablet_manager.cc:1939] T 22f6cbcc079248b08733cfb9c01e847f P 5d9edbc7c90441df9a91ff3b016e0bb3: Deleting consensus metadata
I20260928 12:58:16.648528 12410 catalog_manager.cc:4841] TS 95b8aa3082fa4d5d905364b812035fd4 (127.11.11.67:40371): tablet 22f6cbcc079248b08733cfb9c01e847f (table test-table [id=d57a85c8b0304d068f440a218ce60bd5]) successfully deleted
I20260928 12:58:16.648936 12955 ts_tablet_manager.cc:1905] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:16.650125 12411 catalog_manager.cc:4841] TS 5d9edbc7c90441df9a91ff3b016e0bb3 (127.11.11.65:38693): tablet 22f6cbcc079248b08733cfb9c01e847f (table test-table [id=d57a85c8b0304d068f440a218ce60bd5]) successfully deleted
I20260928 12:58:16.650326 12955 ts_tablet_manager.cc:1918] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:16.650388 12955 log.cc:1196] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/22f6cbcc079248b08733cfb9c01e847f
I20260928 12:58:16.650683 12955 ts_tablet_manager.cc:1939] T 22f6cbcc079248b08733cfb9c01e847f P 65a8c9801a3043f5b26d69481e1211fa: Deleting consensus metadata
I20260928 12:58:16.651014 12411 catalog_manager.cc:4841] TS 65a8c9801a3043f5b26d69481e1211fa (127.11.11.66:46697): tablet 22f6cbcc079248b08733cfb9c01e847f (table test-table [id=d57a85c8b0304d068f440a218ce60bd5]) successfully deleted
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:16.662225 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.652311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.11.11.65:38693)
0928 12:58:16.652352 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.652354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.652591 (+ 237us) server_negotiation.cc:407] Connection header received
0928 12:58:16.652653 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.652654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.652664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.652725 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.653608 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.654190 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.654945 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.655241 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.655328 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.658931 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.658940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.658941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.658948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.660957 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.661356 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.661359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.661360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.661427 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.661634 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.661635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.661636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.661785 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.661934 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.662017 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.662102 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:58:16.662493 12621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50680
I20260928 12:58:16.669790 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:16.663511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.11.11.126:39549)
0928 12:58:16.663557 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:16.663559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:16.663885 (+ 326us) server_negotiation.cc:407] Connection header received
0928 12:58:16.663935 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:16.663936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:16.663948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:16.663993 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:16.664532 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.665038 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.665722 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:16.665932 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:16.666008 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:16.667171 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:16.667173 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:16.667173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:16.667181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:16.668644 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.669002 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.669003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.669004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.669064 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:16.669277 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:16.669278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:16.669278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:16.669390 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:16.669492 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:16.669610 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 12:58:16.669685 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:58:16.670117 12424 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54884
I20260928 12:58:16.670835 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12496
I20260928 12:58:16.677027 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12636
I20260928 12:58:16.683785 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12776
I20260928 12:58:16.689081 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 12392
2026-09-28T12:58:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2355 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12998](info): set up 2 sockets
Sep 28 12:58:16 dist-test-slave-8sjz krb5kdc[12998](info): commencing operation
krb5kdc: starting...
W20260928 12:58:17.720798 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:17 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:17.839244 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:44233
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:17.923444 13011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:17.923723 13011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:17.923765 13011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:17.924989 13011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:17.925073 13011 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:58:17.925096 13011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:17.925117 13011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:17.925146 13011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:17.925176 13011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:17.926744 13011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:44233
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:17.927042 13011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:17.927289 13011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:17.932374 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:17.932353 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:17.932353 13011 server_base.cc:1034] running on GCE node
W20260928 12:58:17.932395 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:17.932785 13011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:17.933048 13011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:17.934248 13011 hybrid_clock.cc:648] HybridClock initialized: now 1790600297934234 us; error 47 us; skew 500 ppm
Sep 28 12:58:17 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:17.936795 13011 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:17.937361 13011 webserver.cc:460] Webserver started at http://127.11.11.126:38211/ using document root <none> and password file <none>
I20260928 12:58:17.937616 13011 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:17.937686 13011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:17.937885 13011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:17.938982 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "345387211ca2432cbe65904c73a3a6f2"
format_stamp: "Formatted at 2026-09-28 12:58:17 on dist-test-slave-8sjz"
server_key: "0011113a4a5f4d40fc6e3cba02311d59"
server_key_iv: "c56190b698a245343af643d286536366"
server_key_version: "encryptionkey@0"
I20260928 12:58:17.939349 13011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "345387211ca2432cbe65904c73a3a6f2"
format_stamp: "Formatted at 2026-09-28 12:58:17 on dist-test-slave-8sjz"
server_key: "0011113a4a5f4d40fc6e3cba02311d59"
server_key_iv: "c56190b698a245343af643d286536366"
server_key_version: "encryptionkey@0"
I20260928 12:58:17.940801 13011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:17.941550 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:17.941717 13011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:17.941825 13011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "345387211ca2432cbe65904c73a3a6f2"
format_stamp: "Formatted at 2026-09-28 12:58:17 on dist-test-slave-8sjz"
server_key: "0011113a4a5f4d40fc6e3cba02311d59"
server_key_iv: "c56190b698a245343af643d286536366"
server_key_version: "encryptionkey@0"
I20260928 12:58:17.941900 13011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:17.951099 13011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:17.953053 13011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:17.953217 13011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:17.958022 13011 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:44233
I20260928 12:58:17.958081 13088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:44233 every 8 connection(s)
I20260928 12:58:17.958408 13011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:17.958995 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:17.961377 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Bootstrap starting.
I20260928 12:58:17.961975 13089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:17.962267 13089 log.cc:824] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:17.962929 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: No bootstrap required, opened a new log
I20260928 12:58:17.964325 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13011
I20260928 12:58:17.964398 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:17.964568 11309 external_mini_cluster.cc:1428] Setting key 2a3b3b106075676ad6441690281b3773
I20260928 12:58:17.965337 13089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } }
I20260928 12:58:17.965510 13089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:17.965555 13089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345387211ca2432cbe65904c73a3a6f2, State: Initialized, Role: FOLLOWER
I20260928 12:58:17.965727 13089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } }
I20260928 12:58:17.965811 13089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:17.965852 13089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:17.965889 13089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:17.966436 13089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } }
I20260928 12:58:17.966554 13089 leader_election.cc:302] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 345387211ca2432cbe65904c73a3a6f2; no voters:
I20260928 12:58:17.966727 13089 leader_election.cc:288] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:17.966766 13094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:17.966910 13094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 345387211ca2432cbe65904c73a3a6f2, State: Running, Role: LEADER
I20260928 12:58:17.967005 13089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:17.967000 13094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } }
I20260928 12:58:17.967686 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "345387211ca2432cbe65904c73a3a6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } } }
I20260928 12:58:17.967753 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:17.967773 13095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 345387211ca2432cbe65904c73a3a6f2. Latest consensus state: current_term: 1 leader_uuid: "345387211ca2432cbe65904c73a3a6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345387211ca2432cbe65904c73a3a6f2" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44233 } } }
I20260928 12:58:17.967859 13095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:17.967952 13103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:17.968505 13103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:17.970672 13103 catalog_manager.cc:1347] Generated new cluster ID: dd2fd99a33b94569a936cc4a8660a994
I20260928 12:58:17.970743 13103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:17 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:17.976770 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:17.965202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.11.11.126:44233)
0928 12:58:17.965411 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:17.965419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:17.965445 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:58:17.965601 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:17.965608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:17.965669 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:17.965841 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:17.966495 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:17.967852 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:17.968988 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:17.969510 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:17.969632 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:17.972586 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:17.972590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:17.972590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:17.972602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:17.974620 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:17.975104 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:17.975107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:17.975107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:17.975239 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:17.975451 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:17.975452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:17.975452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:17.975774 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:17.975938 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:17.976405 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 12:58:17.976552 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 12:58:17.986590 13103 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:17.987336 13103 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:18.001128 13103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Generated new TSK 0
I20260928 12:58:18.001384 13103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:18.031503 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44233
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:18.115665 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:18.115926 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:18.115970 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:18.117231 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:18.117303 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:18.117342 13115 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:58:18.117390 13115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:18.117429 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:18.118956 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:44233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:18.119251 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:18.119491 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:18.124879 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:18.124888 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:18.125027 13115 server_base.cc:1034] running on GCE node
W20260928 12:58:18.124887 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:18.125353 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:18.125603 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:18.126773 13115 hybrid_clock.cc:648] HybridClock initialized: now 1790600298126769 us; error 38 us; skew 500 ppm
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:18.129423 13115 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:18.129925 13115 webserver.cc:460] Webserver started at http://127.11.11.65:46785/ using document root <none> and password file <none>
I20260928 12:58:18.130167 13115 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:18.130237 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:18.130409 13115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:18.131531 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "2f4dbca803e24703bcc793ff1874237e"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "ac8748d81cb878a4ec46f6069eca69ee"
server_key_iv: "e5afeb36ee91815acfe851d08db4a0c3"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.131904 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "2f4dbca803e24703bcc793ff1874237e"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "ac8748d81cb878a4ec46f6069eca69ee"
server_key_iv: "e5afeb36ee91815acfe851d08db4a0c3"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.133303 13115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:18.134105 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.134313 13115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:18.134411 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "2f4dbca803e24703bcc793ff1874237e"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "ac8748d81cb878a4ec46f6069eca69ee"
server_key_iv: "e5afeb36ee91815acfe851d08db4a0c3"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.134495 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:18.144085 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:18.146453 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:18.146669 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:18.147290 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:18.147341 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.147374 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:18.147393 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:18.154712 13115 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38631
I20260928 12:58:18.154852 13245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38631 every 8 connection(s)
I20260928 12:58:18.155124 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:18.156801 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13115
I20260928 12:58:18.156910 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:18.157097 11309 external_mini_cluster.cc:1428] Setting key 86ad62f23692528ec66cdc2cb4e043c4
I20260928 12:58:18.157436 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.147750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:37211 (local address 127.11.11.126:44233)
0928 12:58:18.147827 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.147830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.148095 (+ 265us) server_negotiation.cc:407] Connection header received
0928 12:58:18.148864 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.148866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.148879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.148940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.149896 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.150414 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.151449 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.151665 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.151744 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.154355 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.154357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.154357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.154366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.155982 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.156427 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.156429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.156429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.156482 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.156694 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.156695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.156696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.156837 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.156926 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.157240 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.157346 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:58:18.157605 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.147752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.65:37211)
0928 12:58:18.147931 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.147989 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.148730 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.149026 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.149030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.149449 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.149788 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.149796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.150524 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.150525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.151354 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.151356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.151759 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.151761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.151892 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.152455 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.152463 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.154255 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.156110 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.156115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.156117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.156360 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.156525 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.156525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.156526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.156618 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.156915 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.156921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.157084 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.157196 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.157368 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260928 12:58:18.158147 13246 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44233
I20260928 12:58:18.158247 13246 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:18.158439 13246 heartbeater.cc:511] Master 127.11.11.126:44233 requested a full tablet report, sending...
I20260928 12:58:18.158969 13042 ts_manager.cc:194] Registered new tserver with Master: 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631)
I20260928 12:58:18.159914 13042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:37211
I20260928 12:58:18.164944 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.159222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47461 (local address 127.11.11.126:44233)
0928 12:58:18.159259 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.159261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.159406 (+ 145us) server_negotiation.cc:407] Connection header received
0928 12:58:18.159480 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.159481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.159491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.159546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.159887 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.160474 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.161212 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.161421 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.161504 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.162399 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.162400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.162401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.162408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.163760 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.164138 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.164139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.164140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.164203 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.164419 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.164420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.164420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.164536 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.164612 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.164765 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.164848 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:18.164948 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.159181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.65:47461)
0928 12:58:18.159331 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.159341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.159421 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.159563 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.159564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.159735 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.159805 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.159807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.160570 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.160571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.161112 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.161113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.161505 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.161506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.161589 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.162026 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.162030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.162297 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.163845 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.163846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.163846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.164022 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.164286 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.164286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.164287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.164346 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.164603 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.164604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.164651 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.164740 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.164851 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":7,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:18.183545 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44233
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:18.266379 13255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:18.266701 13255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:18.266757 13255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:18.268221 13255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:18.268287 13255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:18.268326 13255 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:58:18.268373 13255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:18.268414 13255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:18.269971 13255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:44233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:18.270229 13255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:18.270483 13255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:18.275487 13261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:18.275538 13260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:18.275532 13263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:18.275781 13255 server_base.cc:1034] running on GCE node
I20260928 12:58:18.276002 13255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:18.276276 13255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:18.277453 13255 hybrid_clock.cc:648] HybridClock initialized: now 1790600298277430 us; error 28 us; skew 500 ppm
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:18.280144 13255 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:18.280606 13255 webserver.cc:460] Webserver started at http://127.11.11.66:40149/ using document root <none> and password file <none>
I20260928 12:58:18.280819 13255 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:18.280861 13255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:18.280954 13255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:18.281980 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "3875b43f25de4f15a4394b8f43e59103"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "e591e55f7a1aef7060f71f805ee7ee15"
server_key_iv: "df914482c4905544f55acb91e6e57943"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.282269 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "3875b43f25de4f15a4394b8f43e59103"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "e591e55f7a1aef7060f71f805ee7ee15"
server_key_iv: "df914482c4905544f55acb91e6e57943"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.283560 13255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:18.284372 13270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.284545 13255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.284639 13255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "3875b43f25de4f15a4394b8f43e59103"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "e591e55f7a1aef7060f71f805ee7ee15"
server_key_iv: "df914482c4905544f55acb91e6e57943"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.284744 13255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:18.297366 13255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:18.300117 13255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:18.300251 13255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:18.300763 13255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:18.300801 13255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.300827 13255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:18.300894 13255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.308909 13255 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:36779
I20260928 12:58:18.308959 13385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:36779 every 8 connection(s)
I20260928 12:58:18.309317 13255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:18.312906 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.301519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:51471 (local address 127.11.11.126:44233)
0928 12:58:18.301577 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.301580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.301879 (+ 299us) server_negotiation.cc:407] Connection header received
0928 12:58:18.302505 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.302506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.302520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.302592 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.303703 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.304683 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.306597 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.306865 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.306931 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.309930 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.309931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.309932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.309941 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.311406 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.311842 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.311843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.311844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.311908 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.312231 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.312232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.312233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.312361 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.312450 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.312729 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.312814 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:58:18.313107 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.301525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.66:51471)
0928 12:58:18.301733 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.301774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.302378 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.302728 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.302734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.303220 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.303563 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.303571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.304800 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.304801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.306470 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.306474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.306965 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.306966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.307171 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.307884 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.307897 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.309838 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.311516 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.311523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.311525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.311761 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.311986 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.311987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.311987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.312160 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.312438 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.312444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.312591 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.312705 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.312871 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260928 12:58:18.313654 13386 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44233
I20260928 12:58:18.313746 13386 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:18.313943 13386 heartbeater.cc:511] Master 127.11.11.126:44233 requested a full tablet report, sending...
I20260928 12:58:18.314363 13042 ts_manager.cc:194] Registered new tserver with Master: 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779)
I20260928 12:58:18.314962 13042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:51471
I20260928 12:58:18.318754 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13255
I20260928 12:58:18.318861 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:18.319093 11309 external_mini_cluster.cc:1428] Setting key cfbbcf755030c55a4add35aa74cdc43f
I20260928 12:58:18.321002 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.314855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:48487 (local address 127.11.11.126:44233)
0928 12:58:18.314898 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.314900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.314929 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:58:18.314998 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.314999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.315007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.315087 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.315602 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.316135 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.316857 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.317036 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.317090 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.317983 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.317985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.317985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.317991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.319635 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.320069 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.320070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.320071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.320138 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.320385 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.320386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.320386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.320553 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.320651 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.320816 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.320901 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:58:18.321271 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.314748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.66:48487)
0928 12:58:18.314878 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.314889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.314942 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.315199 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.315200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.315447 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.315510 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.315513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.316248 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.316249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.316781 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.316782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.317106 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.317106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.317179 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.317595 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.317600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.317897 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.319718 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.319719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.319738 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.319956 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.320238 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.320238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.320239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.320313 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.320634 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.320635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.320686 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.320787 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.320902 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:18.346503 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44233
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:18.434803 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:18.435050 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:18.435079 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:18.436534 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:18.436582 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:18.436606 13395 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:58:18.436666 13395 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:18.436695 13395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:18.438386 13395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:44233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:18.438678 13395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:18.438963 13395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:18.444379 13403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:18.444391 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:18.444391 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:18.444592 13395 server_base.cc:1034] running on GCE node
I20260928 12:58:18.444869 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:18.445096 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:18.446238 13395 hybrid_clock.cc:648] HybridClock initialized: now 1790600298446228 us; error 29 us; skew 500 ppm
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:18.448823 13395 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:18.449295 13395 webserver.cc:460] Webserver started at http://127.11.11.67:35667/ using document root <none> and password file <none>
I20260928 12:58:18.449522 13395 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:18.449568 13395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:18.449710 13395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:18.450796 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "04d483a68d204cf8b1a9af1c2152af16"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "f1762234ad06c07f9eef1bd779516d44"
server_key_iv: "2db6bc156202e8ad0e7179c5014261e7"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.451184 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "04d483a68d204cf8b1a9af1c2152af16"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "f1762234ad06c07f9eef1bd779516d44"
server_key_iv: "2db6bc156202e8ad0e7179c5014261e7"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.452873 13395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:18.453660 13410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.453843 13395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.453949 13395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "04d483a68d204cf8b1a9af1c2152af16"
format_stamp: "Formatted at 2026-09-28 12:58:18 on dist-test-slave-8sjz"
server_key: "f1762234ad06c07f9eef1bd779516d44"
server_key_iv: "2db6bc156202e8ad0e7179c5014261e7"
server_key_version: "encryptionkey@0"
I20260928 12:58:18.454025 13395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:18.467330 13395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:18.469719 13395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:18.469890 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:18.470415 13395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:18.470479 13395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.470530 13395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:18.470556 13395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.478128 13395 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35687
I20260928 12:58:18.478245 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35687 every 8 connection(s)
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:18.478552 13395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:18.481603 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.471078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:44079 (local address 127.11.11.126:44233)
0928 12:58:18.471125 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.471127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.471381 (+ 254us) server_negotiation.cc:407] Connection header received
0928 12:58:18.472090 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.472091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.472105 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.472170 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.473072 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.473875 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.475529 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.475796 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.475889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.479060 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.479062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.479062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.479070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.480356 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.480779 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.480781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.480781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.480870 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.481082 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.481082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.481082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.481202 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.481256 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.481472 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.481538 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:58:18.481777 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.471089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.67:44079)
0928 12:58:18.471268 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.471306 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.471946 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.472252 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.472256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.472665 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.472974 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.472980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.473959 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.473961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.475448 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.475451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.475899 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.475900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.476085 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.476876 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.476888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.478950 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.480430 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.480436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.480438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.480660 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.480937 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.480938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.480938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.481031 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.481258 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.481263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.481378 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.481460 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.481579 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260928 12:58:18.482136 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13395
I20260928 12:58:18.482227 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:18.482470 11309 external_mini_cluster.cc:1428] Setting key db5c081e872cea55b4c531fd537b476e
I20260928 12:58:18.483816 13526 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44233
I20260928 12:58:18.483914 13526 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:18.484182 13526 heartbeater.cc:511] Master 127.11.11.126:44233 requested a full tablet report, sending...
I20260928 12:58:18.484560 13043 ts_manager.cc:194] Registered new tserver with Master: 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:18.484959 13043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:44079
I20260928 12:58:18.489676 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.483884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.11.11.67:59447)
0928 12:58:18.484008 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.484021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.484103 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.484222 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.484223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.484468 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.484533 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.484535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.485295 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.485297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.485878 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.485880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.486272 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.486273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.486353 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.486784 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.486788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.487066 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.488592 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.488593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.488594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.488790 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.489015 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.489016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.489017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.489081 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.489336 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.489338 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.489383 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.489472 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.489576 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260928 12:58:18.489674 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.483965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:59447 (local address 127.11.11.126:44233)
0928 12:58:18.484011 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.484013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.484087 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:58:18.484159 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.484160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.484169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.484210 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.484630 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.485187 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.485979 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.486183 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.486270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.487172 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.487174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.487175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.487182 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.488508 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.488880 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.488881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.488882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.488942 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.489153 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.489154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.489154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.489270 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.489345 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.489495 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.489574 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260928 12:58:18.494308 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:18.517457 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.510039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.11.11.126:44233)
0928 12:58:18.510120 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.510122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.510131 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:18.510159 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.510160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.510169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.510221 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.510785 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.511280 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.512055 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.512308 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.512375 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.514652 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.514653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.514654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.514661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.516443 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.516790 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.516791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.516792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.516854 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.517024 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.517025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.517026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.517130 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.517215 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.517325 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.517387 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:58:18.519176 13043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:18.521020 13043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:18.527985 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.523652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35687 (local address 127.0.0.1:35492)
0928 12:58:18.524090 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.524100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.524141 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.524335 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.524336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.524337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.524433 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.524435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525879 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.525880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.526817 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.526819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527648 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.527650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.527728 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.527743 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.527816 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.527873 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":93,"threads_started":1}
I20260928 12:58:18.528090 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.523712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.11.11.66:36779)
0928 12:58:18.523926 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.523930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.523940 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:18.523998 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.524001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.524018 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.524150 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.524448 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525711 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.526946 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527703 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.527783 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.527839 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.527913 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.527950 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260928 12:58:18.528155 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.523639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38631 (local address 127.0.0.1:59432)
0928 12:58:18.523898 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.523908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.523956 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.524450 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.524451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.524454 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.524513 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.524515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525414 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.525415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.526844 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.526846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527802 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.527804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.527869 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.527873 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.527907 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.527954 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260928 12:58:18.528374 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.523433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.0.0.1:53284)
0928 12:58:18.523530 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.523662 (+ 132us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.523734 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.524238 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.524241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.524244 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.524344 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.524347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525803 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.525804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.526843 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.526846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527788 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.527788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.527846 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.527849 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.527899 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.527935 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":7,"thread_start_us":54,"threads_started":1}
I20260928 12:58:18.528425 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.523877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.11.11.67:35687)
0928 12:58:18.524082 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.524157 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.524169 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:18.524214 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.524217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.524233 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.524513 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.524574 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525798 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.526903 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527552 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.527649 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.527716 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.528262 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.528306 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260928 12:58:18.528879 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.524142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.11.11.65:38631)
0928 12:58:18.524278 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.524283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.524292 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:18.524332 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.524334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.524351 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.524589 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.524661 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.525316 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.526946 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.527711 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.528054 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.528631 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.528741 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.528785 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260928 12:58:18.529083 13320 tablet_service.cc:1467] Processing CreateTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e (DEFAULT_TABLE table=test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.529101 13460 tablet_service.cc:1467] Processing CreateTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e (DEFAULT_TABLE table=test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.529469 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a526cebbfb44b786f7e43e63e67e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.529472 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a526cebbfb44b786f7e43e63e67e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.529421 13180 tablet_service.cc:1467] Processing CreateTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e (DEFAULT_TABLE table=test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.529712 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a526cebbfb44b786f7e43e63e67e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.532313 13549 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Bootstrap starting.
I20260928 12:58:18.533237 13551 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Bootstrap starting.
I20260928 12:58:18.533566 13549 tablet_bootstrap.cc:654] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.533929 13549 log.cc:824] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:18.534003 13551 tablet_bootstrap.cc:654] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.534312 13551 log.cc:824] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:18.534655 13550 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Bootstrap starting.
I20260928 12:58:18.534824 13549 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: No bootstrap required, opened a new log
I20260928 12:58:18.534910 13549 ts_tablet_manager.cc:1397] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:58:18.535216 13551 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: No bootstrap required, opened a new log
I20260928 12:58:18.535269 13550 tablet_bootstrap.cc:654] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.535327 13551 ts_tablet_manager.cc:1397] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:18.535593 13550 log.cc:824] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:18.536296 13550 tablet_bootstrap.cc:492] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: No bootstrap required, opened a new log
I20260928 12:58:18.536379 13550 ts_tablet_manager.cc:1397] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:18.537688 13551 raft_consensus.cc:348] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.537652 13549 raft_consensus.cc:348] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.537838 13551 raft_consensus.cc:374] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.537838 13549 raft_consensus.cc:374] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.537882 13549 raft_consensus.cc:729] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.537882 13551 raft_consensus.cc:729] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.538050 13551 consensus_queue.cc:260] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.538077 13549 consensus_queue.cc:260] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.538098 13550 raft_consensus.cc:348] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.538264 13550 raft_consensus.cc:374] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.538313 13550 raft_consensus.cc:729] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.538354 13551 ts_tablet_manager.cc:1428] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:18.538354 13549 ts_tablet_manager.cc:1428] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:18.538435 13550 consensus_queue.cc:260] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.538484 13526 heartbeater.cc:503] Master 127.11.11.126:44233 was elected leader, sending a full tablet report...
I20260928 12:58:18.538484 13246 heartbeater.cc:503] Master 127.11.11.126:44233 was elected leader, sending a full tablet report...
I20260928 12:58:18.538690 13550 ts_tablet_manager.cc:1428] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:58:18.538698 13386 heartbeater.cc:503] Master 127.11.11.126:44233 was elected leader, sending a full tablet report...
W20260928 12:58:18.560072 13387 tablet.cc:2369] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:18.625638 13556 raft_consensus.cc:482] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:18.625778 13556 raft_consensus.cc:504] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.626145 13556 leader_election.cc:288] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631), 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779)
I20260928 12:58:18.629073 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.626336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.11.11.67:44205)
0928 12:58:18.626394 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.626407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.626454 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.626626 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.626627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.626629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.626705 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.626706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.627390 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.627391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.628212 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.628214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.628798 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.628886 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.628890 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.628954 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.628994 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260928 12:58:18.629073 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.626413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:44205 (local address 127.11.11.66:36779)
0928 12:58:18.626469 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.626471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.626482 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:18.626525 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.626526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.626536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.626606 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.626801 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.627301 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628311 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.628693 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628796 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.628806 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.628972 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.629011 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:18.629200 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.626441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:49177 (local address 127.11.11.65:38631)
0928 12:58:18.626491 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.626493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.626683 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:58:18.626755 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.626756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.626767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.626868 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.627058 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.627521 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628454 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.628832 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628916 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.628924 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.629095 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.629142 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:18.629197 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.626343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38631 (local address 127.11.11.67:49177)
0928 12:58:18.626586 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.626596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.626693 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.626873 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.626874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.626875 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.626984 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.626985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.627613 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.627614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.628362 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.628364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.628926 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.628927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.629011 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.629014 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.629076 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.629115 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":74,"thread_start_us":118,"threads_started":1}
I20260928 12:58:18.629462 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a526cebbfb44b786f7e43e63e67e6e" candidate_uuid: "04d483a68d204cf8b1a9af1c2152af16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103" is_pre_election: true
I20260928 12:58:18.629522 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a526cebbfb44b786f7e43e63e67e6e" candidate_uuid: "04d483a68d204cf8b1a9af1c2152af16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4dbca803e24703bcc793ff1874237e" is_pre_election: true
I20260928 12:58:18.629650 13340 raft_consensus.cc:2454] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04d483a68d204cf8b1a9af1c2152af16 in term 0.
I20260928 12:58:18.629675 13200 raft_consensus.cc:2454] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04d483a68d204cf8b1a9af1c2152af16 in term 0.
I20260928 12:58:18.629932 13413 leader_election.cc:302] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04d483a68d204cf8b1a9af1c2152af16, 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:18.630057 13556 raft_consensus.cc:2781] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:18.630113 13556 raft_consensus.cc:482] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:18.630136 13556 raft_consensus.cc:3037] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.630820 13556 raft_consensus.cc:504] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.630964 13556 leader_election.cc:288] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631), 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779)
I20260928 12:58:18.631146 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a526cebbfb44b786f7e43e63e67e6e" candidate_uuid: "04d483a68d204cf8b1a9af1c2152af16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103"
I20260928 12:58:18.631145 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a526cebbfb44b786f7e43e63e67e6e" candidate_uuid: "04d483a68d204cf8b1a9af1c2152af16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4dbca803e24703bcc793ff1874237e"
I20260928 12:58:18.631237 13340 raft_consensus.cc:3037] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.631250 13200 raft_consensus.cc:3037] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.632169 13340 raft_consensus.cc:2454] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04d483a68d204cf8b1a9af1c2152af16 in term 1.
I20260928 12:58:18.632169 13200 raft_consensus.cc:2454] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04d483a68d204cf8b1a9af1c2152af16 in term 1.
I20260928 12:58:18.632385 13411 leader_election.cc:302] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04d483a68d204cf8b1a9af1c2152af16, 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:18.632498 13556 raft_consensus.cc:2781] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:18.632725 13556 raft_consensus.cc:686] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 1 LEADER]: Becoming Leader. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Running, Role: LEADER
I20260928 12:58:18.632871 13556 consensus_queue.cc:237] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.633744 13042 catalog_manager.cc:5495] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "04d483a68d204cf8b1a9af1c2152af16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } health_report { overall_health: HEALTHY } } }
W20260928 12:58:18.656598 13247 tablet.cc:2369] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 12:58:18 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:18.667940 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.660867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.11.11.126:44233)
0928 12:58:18.660937 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.660939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.661068 (+ 129us) server_negotiation.cc:407] Connection header received
0928 12:58:18.661115 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.661115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.661125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.661189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:18.661666 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.662146 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.662855 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.663035 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.663093 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.665304 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:18.665310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:18.665310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:18.665317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:18.666685 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.667073 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.667074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.667075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.667130 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:18.667370 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:18.667371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:18.667371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:18.667478 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:18.667555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.667736 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.667824 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:18.668064 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.660811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44233 (local address 127.0.0.1:47376)
0928 12:58:18.660986 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.660999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.661070 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.661205 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.661205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.661545 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:18.661594 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.661597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.662215 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.662216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.662767 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.662769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.663131 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.663132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.663183 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.663717 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:18.663722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:18.665199 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:18.666776 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.666785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.666787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.666983 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.667196 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:18.667197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:18.667197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:18.667307 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:18.667546 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:18.667554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:18.667600 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.667712 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.667898 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":78,"thread_start_us":115,"threads_started":1}
I20260928 12:58:18.669984 13042 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:47376:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:58:18.670236 13042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:18.671767 13180 tablet_service.cc:1467] Processing CreateTablet for tablet 7728ab2280a74a5888339dce13d890f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=837f88dc25d4441daa595882f434fd45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:18.671797 13320 tablet_service.cc:1467] Processing CreateTablet for tablet 7728ab2280a74a5888339dce13d890f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=837f88dc25d4441daa595882f434fd45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:18.671954 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7728ab2280a74a5888339dce13d890f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.671970 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7728ab2280a74a5888339dce13d890f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.671981 13460 tablet_service.cc:1467] Processing CreateTablet for tablet 7728ab2280a74a5888339dce13d890f3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=837f88dc25d4441daa595882f434fd45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:18.672142 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7728ab2280a74a5888339dce13d890f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.673156 13551 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: Bootstrap starting.
I20260928 12:58:18.673519 13551 tablet_bootstrap.cc:654] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.673619 13550 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: Bootstrap starting.
I20260928 12:58:18.673620 13549 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: Bootstrap starting.
I20260928 12:58:18.674055 13549 tablet_bootstrap.cc:654] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.674371 13551 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: No bootstrap required, opened a new log
I20260928 12:58:18.674376 13550 tablet_bootstrap.cc:654] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.674423 13551 ts_tablet_manager.cc:1397] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.674583 13551 raft_consensus.cc:348] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.674680 13551 raft_consensus.cc:374] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.674705 13551 raft_consensus.cc:729] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.674687 13549 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: No bootstrap required, opened a new log
I20260928 12:58:18.674760 13549 ts_tablet_manager.cc:1397] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.674754 13551 consensus_queue.cc:260] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.674868 13551 ts_tablet_manager.cc:1428] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.674902 13555 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.674908 13549 raft_consensus.cc:348] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.675001 13555 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.675017 13549 raft_consensus.cc:374] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.675040 13549 raft_consensus.cc:729] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.675096 13549 consensus_queue.cc:260] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.675204 13549 ts_tablet_manager.cc:1428] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.675223 13550 tablet_bootstrap.cc:492] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: No bootstrap required, opened a new log
I20260928 12:58:18.675267 13550 ts_tablet_manager.cc:1397] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:58:18.675249 13556 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.675347 13556 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.675410 13550 raft_consensus.cc:348] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.675485 13550 raft_consensus.cc:374] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.675514 13550 raft_consensus.cc:729] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.675562 13550 consensus_queue.cc:260] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.675661 13550 ts_tablet_manager.cc:1428] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:18.675894 13557 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.676021 13557 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 12:58:18.729207 13527 tablet.cc:2369] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:18.812209 13557 raft_consensus.cc:482] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:18.812325 13557 raft_consensus.cc:504] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.812645 13557 leader_election.cc:288] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:18.815856 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.813004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:54587 (local address 127.11.11.65:38631)
0928 12:58:18.813052 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.813054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.813175 (+ 121us) server_negotiation.cc:407] Connection header received
0928 12:58:18.813207 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.813208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.813218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.813285 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.813534 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.814050 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815096 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.815459 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815517 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.815524 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.815732 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.815778 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:58:18.815871 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.813018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:50981 (local address 127.11.11.67:35687)
0928 12:58:18.813060 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.813062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.813070 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:18.813108 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.813109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.813116 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.813214 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:18.813434 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.813954 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815133 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.815510 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815589 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.815598 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.815764 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.815809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:18.815865 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.812917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35687 (local address 127.11.11.66:50981)
0928 12:58:18.812982 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.812994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.813041 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.813246 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.813247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.813267 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.813348 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.813350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.814045 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.814046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.815014 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.815017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.815602 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.815665 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.815668 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.815743 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.815781 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":35}
I20260928 12:58:18.815882 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.812917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38631 (local address 127.11.11.66:54587)
0928 12:58:18.813086 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.813094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.813149 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.813306 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.813307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.813309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:18.813442 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.813444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.814147 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.814148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.815004 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.815007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.815559 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.815561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.815636 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.815642 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.815695 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.815751 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":19,"thread_start_us":76,"threads_started":1}
I20260928 12:58:18.816208 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7728ab2280a74a5888339dce13d890f3" candidate_uuid: "3875b43f25de4f15a4394b8f43e59103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4dbca803e24703bcc793ff1874237e" is_pre_election: true
I20260928 12:58:18.816253 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7728ab2280a74a5888339dce13d890f3" candidate_uuid: "3875b43f25de4f15a4394b8f43e59103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16" is_pre_election: true
I20260928 12:58:18.816325 13200 raft_consensus.cc:2454] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3875b43f25de4f15a4394b8f43e59103 in term 0.
I20260928 12:58:18.816351 13463 raft_consensus.cc:2454] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3875b43f25de4f15a4394b8f43e59103 in term 0.
I20260928 12:58:18.816495 13273 leader_election.cc:302] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:18.816725 13557 raft_consensus.cc:2781] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:18.816762 13557 raft_consensus.cc:482] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:18.816777 13557 raft_consensus.cc:3037] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.817456 13557 raft_consensus.cc:504] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.817629 13557 leader_election.cc:288] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:18.817811 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7728ab2280a74a5888339dce13d890f3" candidate_uuid: "3875b43f25de4f15a4394b8f43e59103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16"
I20260928 12:58:18.817819 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7728ab2280a74a5888339dce13d890f3" candidate_uuid: "3875b43f25de4f15a4394b8f43e59103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4dbca803e24703bcc793ff1874237e"
I20260928 12:58:18.817911 13200 raft_consensus.cc:3037] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.817909 13463 raft_consensus.cc:3037] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:18.818643 13200 raft_consensus.cc:2454] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3875b43f25de4f15a4394b8f43e59103 in term 1.
I20260928 12:58:18.818781 13463 raft_consensus.cc:2454] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3875b43f25de4f15a4394b8f43e59103 in term 1.
I20260928 12:58:18.818825 13273 leader_election.cc:302] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:18.818898 13557 raft_consensus.cc:2781] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:18.819042 13557 raft_consensus.cc:686] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [term 1 LEADER]: Becoming Leader. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Running, Role: LEADER
I20260928 12:58:18.819156 13557 consensus_queue.cc:237] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.819629 13571 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: TxnStatusTablet state changed. Reason: New leader 3875b43f25de4f15a4394b8f43e59103. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.819747 13571 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:18.819936 13573 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:58:18.819916 13043 catalog_manager.cc:5495] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66). New cstate: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:18.820075 13573 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:58:18.860183 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.857148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.0.0.1:53286)
0928 12:58:18.857241 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.857253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.857296 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.857502 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.857503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.857505 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:18.857562 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.857564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.858273 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.858274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.859250 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.859254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.859602 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.859604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.859678 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.859697 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.859953 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.859961 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.860017 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.860090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 12:58:18.860189 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.857230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.11.11.66:36779)
0928 12:58:18.857290 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.857292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.857303 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:18.857375 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.857376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.857392 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.857490 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:18.857628 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.858194 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.859332 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.859517 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.859575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.859792 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.859891 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.859951 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.860018 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.860051 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:18.862176 13200 raft_consensus.cc:1261] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Refusing update from remote peer 3875b43f25de4f15a4394b8f43e59103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:18.862156 13463 raft_consensus.cc:1261] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Refusing update from remote peer 3875b43f25de4f15a4394b8f43e59103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:18.862515 13571 consensus_queue.cc:1035] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:18.862658 13557 consensus_queue.cc:1035] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:18.863775 13556 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: TxnStatusTablet state changed. Reason: New leader 3875b43f25de4f15a4394b8f43e59103. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.863871 13556 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.864686 13557 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.864768 13557 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:18.865159 13555 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: TxnStatusTablet state changed. Reason: New leader 3875b43f25de4f15a4394b8f43e59103. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.865280 13555 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.865608 13556 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.865686 13556 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 04d483a68d204cf8b1a9af1c2152af16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.865744 13576 mvcc.cc:205] Tried to move back new op lower bound from 7334298824136830976 to 7334298823963824128. Current Snapshot: MvccSnapshot[applied={T|T < 7334298824136830976 or (T in {7334298824136830976})}]
I20260928 12:58:18.866266 13577 mvcc.cc:205] Tried to move back new op lower bound from 7334298824136830976 to 7334298823963824128. Current Snapshot: MvccSnapshot[applied={T|T < 7334298824136830976 or (T in {7334298824136830976})}]
I20260928 12:58:18.867313 13571 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.867400 13571 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 3875b43f25de4f15a4394b8f43e59103: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:18.867972 13555 tablet_replica.cc:439] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3875b43f25de4f15a4394b8f43e59103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } } }
I20260928 12:58:18.868076 13555 tablet_replica.cc:442] T 7728ab2280a74a5888339dce13d890f3 P 2f4dbca803e24703bcc793ff1874237e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:18.868569 13580 mvcc.cc:205] Tried to move back new op lower bound from 7334298824136830976 to 7334298823963824128. Current Snapshot: MvccSnapshot[applied={T|T < 7334298824136830976 or (T in {7334298824136830976})}]
I20260928 12:58:18.870361 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.867450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.11.11.67:35687)
0928 12:58:18.867600 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.867602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.867609 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:18.867638 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.867638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.867656 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.867718 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:18.867968 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.868754 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.869547 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.869850 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.869925 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.870047 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.870175 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.870254 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.870271 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.870294 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260928 12:58:18.876773 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.873076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:57933 (local address 127.11.11.66:36779)
0928 12:58:18.873887 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.873889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.873899 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:18.873923 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.873925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.873935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.873996 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:18.874204 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.874892 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.876121 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.876319 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.876376 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.876579 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.876604 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.876643 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.876696 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.876712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784}
I20260928 12:58:18.876812 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.873020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.11.11.67:57933)
0928 12:58:18.873097 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.873109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.873151 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.874006 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.874008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.874009 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:18.874096 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.874098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.874989 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.874991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.876002 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.876005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.876394 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.876395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.876463 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.876481 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.876644 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.876646 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.876695 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.876737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:58:18.879487 13200 raft_consensus.cc:1261] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Refusing update from remote peer 04d483a68d204cf8b1a9af1c2152af16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:18.879541 13340 raft_consensus.cc:1261] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Refusing update from remote peer 04d483a68d204cf8b1a9af1c2152af16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:18.879802 13556 consensus_queue.cc:1035] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:18.879976 13556 consensus_queue.cc:1035] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:18.888288 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.885411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49785 (local address 127.11.11.67:35687)
0928 12:58:18.885478 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:18.885480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:18.885617 (+ 137us) server_negotiation.cc:407] Connection header received
0928 12:58:18.885690 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:18.885691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:18.885702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:18.885757 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:18.885895 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.886362 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.887542 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.887748 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.887831 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.888019 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.888077 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.888131 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:18.888195 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 12:58:18.888211 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:58:18.888304 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:18.885373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35687 (local address 127.11.11.66:49785)
0928 12:58:18.885541 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:18.885552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:18.885633 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:18.885760 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:18.885761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:18.885762 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:18.885822 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.885824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.886444 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.886445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.887449 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:18.887452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:18.887832 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:18.887833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:18.887898 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:18.887918 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:18.888138 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:18.888140 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:18.888190 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:58:18.888232 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 12:58:18.897375 13042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:18.897496 13042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:18.898268 13042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Sending DeleteTablet for 3 replicas of tablet c3a526cebbfb44b786f7e43e63e67e6e
I20260928 12:58:18.898581 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59432
I20260928 12:58:18.898658 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:35492
I20260928 12:58:18.898820 13594 tablet_replica.cc:330] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: stopping tablet replica
I20260928 12:58:18.898924 13594 raft_consensus.cc:2229] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:18.898985 13594 raft_consensus.cc:2258] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:18.899351 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a526cebbfb44b786f7e43e63e67e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53284
I20260928 12:58:18.899457 13594 ts_tablet_manager.cc:1905] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:18.899607 13042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:18.899765 13042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:18.901161 13180 tablet_service.cc:1467] Processing CreateTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 (DEFAULT_TABLE table=test-table [id=f593f09cd4734081b4509d420186cb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.901275 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e111dbdb634e039d4a3bcc71b01e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.901719 13594 ts_tablet_manager.cc:1918] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:18.901784 13594 log.cc:1196] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/c3a526cebbfb44b786f7e43e63e67e6e
I20260928 12:58:18.902048 13594 ts_tablet_manager.cc:1939] T c3a526cebbfb44b786f7e43e63e67e6e P 2f4dbca803e24703bcc793ff1874237e: Deleting consensus metadata
I20260928 12:58:18.902470 13029 catalog_manager.cc:4841] TS 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631): tablet c3a526cebbfb44b786f7e43e63e67e6e (table test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]) successfully deleted
I20260928 12:58:18.902525 13551 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Bootstrap starting.
I20260928 12:58:18.902917 13551 tablet_bootstrap.cc:654] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.903407 13460 tablet_service.cc:1467] Processing CreateTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 (DEFAULT_TABLE table=test-table [id=f593f09cd4734081b4509d420186cb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.903517 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e111dbdb634e039d4a3bcc71b01e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.903550 13551 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: No bootstrap required, opened a new log
I20260928 12:58:18.903609 13551 ts_tablet_manager.cc:1397] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.903745 13551 raft_consensus.cc:348] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.903812 13551 raft_consensus.cc:374] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.903833 13551 raft_consensus.cc:729] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.903877 13551 consensus_queue.cc:260] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.903963 13551 ts_tablet_manager.cc:1428] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:18.904881 13549 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Bootstrap starting.
I20260928 12:58:18.905315 13549 tablet_bootstrap.cc:654] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.905947 13320 tablet_service.cc:1467] Processing CreateTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 (DEFAULT_TABLE table=test-table [id=f593f09cd4734081b4509d420186cb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:18.905982 13549 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: No bootstrap required, opened a new log
I20260928 12:58:18.906026 13549 ts_tablet_manager.cc:1397] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:18.906069 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e111dbdb634e039d4a3bcc71b01e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:18.906183 13549 raft_consensus.cc:348] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.906248 13549 raft_consensus.cc:374] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.906270 13549 raft_consensus.cc:729] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.906313 13549 consensus_queue.cc:260] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.906397 13549 ts_tablet_manager.cc:1428] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:18.907399 13550 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Bootstrap starting.
I20260928 12:58:18.907968 13550 tablet_bootstrap.cc:654] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:18.909073 13550 tablet_bootstrap.cc:492] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: No bootstrap required, opened a new log
I20260928 12:58:18.909126 13550 ts_tablet_manager.cc:1397] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:18.909312 13595 tablet_replica.cc:330] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: stopping tablet replica
I20260928 12:58:18.909292 13550 raft_consensus.cc:348] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.909360 13550 raft_consensus.cc:374] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:18.909390 13550 raft_consensus.cc:729] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Initialized, Role: FOLLOWER
I20260928 12:58:18.909442 13550 consensus_queue.cc:260] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:18.909529 13550 ts_tablet_manager.cc:1428] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:18.910403 13595 raft_consensus.cc:2229] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:18.910840 13595 raft_consensus.cc:2258] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:18.911134 13596 tablet_replica.cc:330] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: stopping tablet replica
I20260928 12:58:18.911279 13596 raft_consensus.cc:2229] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:18.911401 13596 raft_consensus.cc:2258] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:18.911455 13595 ts_tablet_manager.cc:1905] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:18.911912 13596 ts_tablet_manager.cc:1905] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:18.913185 13595 ts_tablet_manager.cc:1918] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:18.913266 13595 log.cc:1196] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/c3a526cebbfb44b786f7e43e63e67e6e
I20260928 12:58:18.913542 13595 ts_tablet_manager.cc:1939] T c3a526cebbfb44b786f7e43e63e67e6e P 04d483a68d204cf8b1a9af1c2152af16: Deleting consensus metadata
I20260928 12:58:18.913998 13028 catalog_manager.cc:4841] TS 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687): tablet c3a526cebbfb44b786f7e43e63e67e6e (table test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]) successfully deleted
I20260928 12:58:18.914019 13596 ts_tablet_manager.cc:1918] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:18.914158 13596 log.cc:1196] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/c3a526cebbfb44b786f7e43e63e67e6e
I20260928 12:58:18.914472 13596 ts_tablet_manager.cc:1939] T c3a526cebbfb44b786f7e43e63e67e6e P 3875b43f25de4f15a4394b8f43e59103: Deleting consensus metadata
I20260928 12:58:18.914891 13027 catalog_manager.cc:4841] TS 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779): tablet c3a526cebbfb44b786f7e43e63e67e6e (table test-table [id=d244f591a7614c3c95fdb69b0cfdb9bc]) successfully deleted
I20260928 12:58:19.059682 13555 raft_consensus.cc:482] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:19.059813 13555 raft_consensus.cc:504] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:19.060253 13555 leader_election.cc:288] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:19.063632 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.060557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35687 (local address 127.11.11.65:37937)
0928 12:58:19.060918 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:19.060929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:19.061010 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:19.061160 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:19.061161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:19.061162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:19.061249 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.061250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.061925 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.061926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.062710 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.062712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.063329 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.063330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.063403 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.063408 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:19.063451 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:58:19.063502 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":195,"thread_start_us":96,"threads_started":1}
I20260928 12:58:19.063637 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.060665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:42397 (local address 127.11.11.66:36779)
0928 12:58:19.060713 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:19.060716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:19.060858 (+ 142us) server_negotiation.cc:407] Connection header received
0928 12:58:19.060941 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:19.060943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:19.060959 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:19.061019 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:19.061247 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.061752 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.062896 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.063242 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.063325 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.063334 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:19.063527 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:58:19.063565 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:19.063637 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.060737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:37937 (local address 127.11.11.67:35687)
0928 12:58:19.060792 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:19.060795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:19.061052 (+ 257us) server_negotiation.cc:407] Connection header received
0928 12:58:19.061082 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:19.061083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:19.061094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:19.061152 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:19.061322 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.061839 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.062814 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.063243 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.063333 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.063341 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:19.063486 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 12:58:19.063530 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:58:19.063660 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.060557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.11.11.65:42397)
0928 12:58:19.060777 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:19.060792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:19.060872 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:19.061040 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:19.061042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:19.061044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:19.061114 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.061116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.061835 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.061836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.062799 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.062802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.063337 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.063338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.063419 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.063422 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:19.063506 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:58:19.063544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":47,"thread_start_us":75,"threads_started":1}
I20260928 12:58:19.063934 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e111dbdb634e039d4a3bcc71b01e62" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103" is_pre_election: true
I20260928 12:58:19.064069 13340 raft_consensus.cc:2454] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:19.064172 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e111dbdb634e039d4a3bcc71b01e62" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16" is_pre_election: true
I20260928 12:58:19.064287 13463 raft_consensus.cc:2454] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:19.064481 13131 leader_election.cc:302] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:19.064615 13555 raft_consensus.cc:2781] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:19.064662 13555 raft_consensus.cc:482] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:19.064690 13555 raft_consensus.cc:3037] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:19.065378 13555 raft_consensus.cc:504] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:19.065531 13555 leader_election.cc:288] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Requested vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:19.065723 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e111dbdb634e039d4a3bcc71b01e62" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16"
I20260928 12:58:19.065806 13463 raft_consensus.cc:3037] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:19.066501 13463 raft_consensus.cc:2454] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:19.066639 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e111dbdb634e039d4a3bcc71b01e62" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103"
I20260928 12:58:19.066680 13132 leader_election.cc:302] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d483a68d204cf8b1a9af1c2152af16, 2f4dbca803e24703bcc793ff1874237e; no voters:
I20260928 12:58:19.066730 13340 raft_consensus.cc:3037] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:19.066789 13555 raft_consensus.cc:2781] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:19.066960 13555 raft_consensus.cc:686] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 1 LEADER]: Becoming Leader. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Running, Role: LEADER
I20260928 12:58:19.067052 13555 consensus_queue.cc:237] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:19.067644 13340 raft_consensus.cc:2454] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:19.067817 13042 catalog_manager.cc:5495] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4dbca803e24703bcc793ff1874237e (127.11.11.65). New cstate: current_term: 1 leader_uuid: "2f4dbca803e24703bcc793ff1874237e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:19.095244 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.092870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.11.11.65:38631)
0928 12:58:19.092925 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:19.092927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:19.092934 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:19.092973 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:19.092974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:19.092993 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:19.093070 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:19.093192 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.093728 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.094521 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.094729 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.094791 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.094939 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:19.095052 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:19.095115 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:19.095161 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:58:19.095178 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:58:19.100593 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.098157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47175 (local address 127.11.11.66:36779)
0928 12:58:19.098200 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:19.098203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:19.098265 (+ 62us) server_negotiation.cc:407] Connection header received
0928 12:58:19.098322 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:19.098323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:19.098332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:19.098383 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:19.098501 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.098991 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.099931 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.100146 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.100217 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.100393 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:19.100420 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:19.100459 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:19.100517 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:58:19.100533 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:19.100633 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.098051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36779 (local address 127.11.11.65:47175)
0928 12:58:19.098206 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:19.098218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:19.098279 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:19.098395 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:19.098396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:19.098398 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:19.098460 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.098461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.099076 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.099077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.099700 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.099702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.100243 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.100244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.100295 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.100309 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:19.100478 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:19.100479 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:19.100514 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 12:58:19.100547 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 12:58:19.102828 13340 raft_consensus.cc:1261] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:19.102865 13463 raft_consensus.cc:1261] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:19.103320 13602 consensus_queue.cc:1035] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:19.103459 13555 consensus_queue.cc:1035] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:19.110760 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.108288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38631 (local address 127.11.11.66:52909)
0928 12:58:19.108335 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:19.108346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:19.108386 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:19.108517 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:19.108518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:19.108519 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:19.108563 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.108565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.109242 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.109243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.109865 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:19.109867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.110311 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.110312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:19.110372 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.110380 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:19.110594 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:19.110597 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:19.110642 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:58:19.110704 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:58:19.110761 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:19.108341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:52909 (local address 127.11.11.65:38631)
0928 12:58:19.108379 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:19.108381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:19.108389 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:19.108443 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:19.108444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:19.108452 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:19.108513 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:19.108652 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.109175 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.109955 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:19.110243 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:19.110312 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:19.110478 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:19.110515 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:19.110566 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:19.110664 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 12:58:19.110690 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 12:58:19.111146 13580 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:58:20.636245 13042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:20.636370 13042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:20.637302 13042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Sending DeleteTablet for 3 replicas of tablet b0e111dbdb634e039d4a3bcc71b01e62
I20260928 12:58:20.637604 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53284
I20260928 12:58:20.637624 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59432
I20260928 12:58:20.637691 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e111dbdb634e039d4a3bcc71b01e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:35492
I20260928 12:58:20.637887 13633 tablet_replica.cc:330] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: stopping tablet replica
I20260928 12:58:20.639493 13633 raft_consensus.cc:2229] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:20.639572 13633 raft_consensus.cc:2258] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:20.639856 13633 ts_tablet_manager.cc:1905] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.641489 13633 ts_tablet_manager.cc:1918] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:58:20.641588 13633 log.cc:1196] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/b0e111dbdb634e039d4a3bcc71b01e62
I20260928 12:58:20.641822 13633 ts_tablet_manager.cc:1939] T b0e111dbdb634e039d4a3bcc71b01e62 P 3875b43f25de4f15a4394b8f43e59103: Deleting consensus metadata
I20260928 12:58:20.642251 13027 catalog_manager.cc:4841] TS 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779): tablet b0e111dbdb634e039d4a3bcc71b01e62 (table test-table [id=f593f09cd4734081b4509d420186cb74]) successfully deleted
I20260928 12:58:20.643476 13632 tablet_replica.cc:330] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: stopping tablet replica
I20260928 12:58:20.643554 13632 raft_consensus.cc:2229] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:20.643692 13632 raft_consensus.cc:2258] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:20.643961 13632 ts_tablet_manager.cc:1905] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.645483 13634 tablet_replica.cc:330] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: stopping tablet replica
I20260928 12:58:20.645576 13634 raft_consensus.cc:2229] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:20.645628 13634 raft_consensus.cc:2258] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:20.645803 13632 ts_tablet_manager.cc:1918] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:58:20.645869 13632 log.cc:1196] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/b0e111dbdb634e039d4a3bcc71b01e62
I20260928 12:58:20.645956 13634 ts_tablet_manager.cc:1905] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.646136 13632 ts_tablet_manager.cc:1939] T b0e111dbdb634e039d4a3bcc71b01e62 P 2f4dbca803e24703bcc793ff1874237e: Deleting consensus metadata
I20260928 12:58:20.646466 13029 catalog_manager.cc:4841] TS 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631): tablet b0e111dbdb634e039d4a3bcc71b01e62 (table test-table [id=f593f09cd4734081b4509d420186cb74]) successfully deleted
I20260928 12:58:20.647912 13634 ts_tablet_manager.cc:1918] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:58:20.647977 13634 log.cc:1196] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/b0e111dbdb634e039d4a3bcc71b01e62
I20260928 12:58:20.648254 13634 ts_tablet_manager.cc:1939] T b0e111dbdb634e039d4a3bcc71b01e62 P 04d483a68d204cf8b1a9af1c2152af16: Deleting consensus metadata
I20260928 12:58:20.648613 13028 catalog_manager.cc:4841] TS 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687): tablet b0e111dbdb634e039d4a3bcc71b01e62 (table test-table [id=f593f09cd4734081b4509d420186cb74]) successfully deleted
Sep 28 12:58:20 dist-test-slave-8sjz krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:58:20 dist-test-slave-8sjz krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600300, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:20.670557 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:20.656470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.11.11.126:44233)
0928 12:58:20.656634 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:20.656636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:20.656645 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:20.656770 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:20.656771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:20.656784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:20.656876 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:20.657460 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:20.657992 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:20.658874 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:20.659076 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:20.659218 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:20.668084 (+ 8866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:20.668087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:20.668088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:20.668097 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:20.669642 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:20.669950 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:20.669950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:20.669951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:20.670003 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:20.670142 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:20.670143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:20.670143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:20.670260 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:20.670332 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:20.670419 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:58:20.670484 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 12:58:20.672250 13042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:20.672429 13042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:20.673844 13320 tablet_service.cc:1467] Processing CreateTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d (DEFAULT_TABLE table=test-table [id=304baf10cd574c4982d97f438ff3f845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.673869 13460 tablet_service.cc:1467] Processing CreateTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d (DEFAULT_TABLE table=test-table [id=304baf10cd574c4982d97f438ff3f845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.674003 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2eb7aaa1204eb291560a9d5aa49f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.674019 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2eb7aaa1204eb291560a9d5aa49f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.675134 13180 tablet_service.cc:1467] Processing CreateTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d (DEFAULT_TABLE table=test-table [id=304baf10cd574c4982d97f438ff3f845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.675151 13644 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Bootstrap starting.
I20260928 12:58:20.675263 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da2eb7aaa1204eb291560a9d5aa49f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.675552 13644 tablet_bootstrap.cc:654] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.675801 13643 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Bootstrap starting.
I20260928 12:58:20.676244 13644 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: No bootstrap required, opened a new log
I20260928 12:58:20.676288 13643 tablet_bootstrap.cc:654] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.676306 13644 ts_tablet_manager.cc:1397] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:20.676431 13644 raft_consensus.cc:348] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.676522 13644 raft_consensus.cc:374] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.676548 13644 raft_consensus.cc:729] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.676606 13644 consensus_queue.cc:260] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.676774 13644 ts_tablet_manager.cc:1428] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:20.676858 13648 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Bootstrap starting.
I20260928 12:58:20.677253 13648 tablet_bootstrap.cc:654] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.677277 13643 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: No bootstrap required, opened a new log
I20260928 12:58:20.677345 13643 ts_tablet_manager.cc:1397] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:20.677460 13643 raft_consensus.cc:348] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.677558 13643 raft_consensus.cc:374] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.677585 13643 raft_consensus.cc:729] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.677631 13643 consensus_queue.cc:260] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.677762 13643 ts_tablet_manager.cc:1428] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:20.677855 13648 tablet_bootstrap.cc:492] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: No bootstrap required, opened a new log
I20260928 12:58:20.677925 13648 ts_tablet_manager.cc:1397] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:20.678047 13648 raft_consensus.cc:348] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.678114 13648 raft_consensus.cc:374] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.678138 13648 raft_consensus.cc:729] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.678184 13648 consensus_queue.cc:260] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.678298 13648 ts_tablet_manager.cc:1428] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:20.682447 13627 raft_consensus.cc:482] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:20.682508 13627 raft_consensus.cc:504] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.682698 13627 leader_election.cc:288] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:20.682847 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2eb7aaa1204eb291560a9d5aa49f0d" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16" is_pre_election: true
I20260928 12:58:20.682873 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2eb7aaa1204eb291560a9d5aa49f0d" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103" is_pre_election: true
I20260928 12:58:20.682935 13463 raft_consensus.cc:2454] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:20.682974 13340 raft_consensus.cc:2454] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:20.683106 13132 leader_election.cc:302] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d483a68d204cf8b1a9af1c2152af16, 2f4dbca803e24703bcc793ff1874237e; no voters:
I20260928 12:58:20.683213 13627 raft_consensus.cc:2781] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:20.683266 13627 raft_consensus.cc:482] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:20.683292 13627 raft_consensus.cc:3037] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.684109 13627 raft_consensus.cc:504] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.684257 13627 leader_election.cc:288] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Requested vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:20.684378 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2eb7aaa1204eb291560a9d5aa49f0d" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103"
I20260928 12:58:20.684386 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da2eb7aaa1204eb291560a9d5aa49f0d" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16"
I20260928 12:58:20.684458 13340 raft_consensus.cc:3037] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.684458 13463 raft_consensus.cc:3037] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.685192 13463 raft_consensus.cc:2454] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:20.685192 13340 raft_consensus.cc:2454] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:20.685415 13131 leader_election.cc:302] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:20.685542 13627 raft_consensus.cc:2781] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:20.685591 13627 raft_consensus.cc:686] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 1 LEADER]: Becoming Leader. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Running, Role: LEADER
I20260928 12:58:20.685642 13627 consensus_queue.cc:237] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.686267 13043 catalog_manager.cc:5495] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4dbca803e24703bcc793ff1874237e (127.11.11.65). New cstate: current_term: 1 leader_uuid: "2f4dbca803e24703bcc793ff1874237e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:20.696985 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:20.694711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.11.11.65:38631)
0928 12:58:20.694868 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:20.694870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:20.694877 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:20.694896 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:20.694897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:20.694907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:20.694964 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:20.695082 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:20.695550 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:20.696337 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:20.696545 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:20.696622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:20.696746 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:20.696774 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:20.696814 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:20.696861 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:58:20.696879 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":112,"threads_started":1}
I20260928 12:58:20.699218 13340 raft_consensus.cc:1261] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:20.699231 13463 raft_consensus.cc:1261] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:20.699441 13623 consensus_queue.cc:1035] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:20.699563 13627 consensus_queue.cc:1035] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:20.709555 13043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:20.709683 13043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:20.710359 13043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Sending DeleteTablet for 3 replicas of tablet da2eb7aaa1204eb291560a9d5aa49f0d
I20260928 12:58:20.710631 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53284
I20260928 12:58:20.710716 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59432
I20260928 12:58:20.710829 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet da2eb7aaa1204eb291560a9d5aa49f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:20 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:35492
I20260928 12:58:20.710846 13632 tablet_replica.cc:330] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: stopping tablet replica
I20260928 12:58:20.710911 13632 raft_consensus.cc:2229] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:20.710930 13634 tablet_replica.cc:330] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: stopping tablet replica
I20260928 12:58:20.710935 13633 tablet_replica.cc:330] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: stopping tablet replica
I20260928 12:58:20.710965 13632 raft_consensus.cc:2258] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:20.710987 13634 raft_consensus.cc:2229] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:20.710994 13633 raft_consensus.cc:2229] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:20.710934 13043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:58:20.711031 13633 raft_consensus.cc:2258] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:20.711035 13634 raft_consensus.cc:2258] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:58:20.711066 13043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:20.711402 13634 ts_tablet_manager.cc:1905] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.711423 13632 ts_tablet_manager.cc:1905] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.711781 13633 ts_tablet_manager.cc:1905] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:20.712976 13634 ts_tablet_manager.cc:1918] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:20.713076 13634 log.cc:1196] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/da2eb7aaa1204eb291560a9d5aa49f0d
I20260928 12:58:20.713296 13634 ts_tablet_manager.cc:1939] T da2eb7aaa1204eb291560a9d5aa49f0d P 04d483a68d204cf8b1a9af1c2152af16: Deleting consensus metadata
I20260928 12:58:20.713477 13632 ts_tablet_manager.cc:1918] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:20.713577 13632 log.cc:1196] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/da2eb7aaa1204eb291560a9d5aa49f0d
I20260928 12:58:20.713809 13632 ts_tablet_manager.cc:1939] T da2eb7aaa1204eb291560a9d5aa49f0d P 2f4dbca803e24703bcc793ff1874237e: Deleting consensus metadata
I20260928 12:58:20.713943 13320 tablet_service.cc:1467] Processing CreateTablet for tablet cf80b629e59942e4a08b1e072e0529e1 (DEFAULT_TABLE table=test-table [id=5804b9a2a6d2435f9736b9ac2c7ee63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.713976 13180 tablet_service.cc:1467] Processing CreateTablet for tablet cf80b629e59942e4a08b1e072e0529e1 (DEFAULT_TABLE table=test-table [id=5804b9a2a6d2435f9736b9ac2c7ee63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.714068 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf80b629e59942e4a08b1e072e0529e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.714109 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf80b629e59942e4a08b1e072e0529e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.714308 13028 catalog_manager.cc:4841] TS 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687): tablet da2eb7aaa1204eb291560a9d5aa49f0d (table test-table [id=304baf10cd574c4982d97f438ff3f845]) successfully deleted
I20260928 12:58:20.715274 13643 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Bootstrap starting.
I20260928 12:58:20.715274 13648 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e: Bootstrap starting.
I20260928 12:58:20.715299 13460 tablet_service.cc:1467] Processing CreateTablet for tablet cf80b629e59942e4a08b1e072e0529e1 (DEFAULT_TABLE table=test-table [id=5804b9a2a6d2435f9736b9ac2c7ee63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:20.714118 13633 ts_tablet_manager.cc:1918] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:20.715399 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf80b629e59942e4a08b1e072e0529e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:20.715409 13633 log.cc:1196] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/da2eb7aaa1204eb291560a9d5aa49f0d
I20260928 12:58:20.715617 13648 tablet_bootstrap.cc:654] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.715617 13643 tablet_bootstrap.cc:654] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.715628 13633 ts_tablet_manager.cc:1939] T da2eb7aaa1204eb291560a9d5aa49f0d P 3875b43f25de4f15a4394b8f43e59103: Deleting consensus metadata
I20260928 12:58:20.715953 13027 catalog_manager.cc:4841] TS 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779): tablet da2eb7aaa1204eb291560a9d5aa49f0d (table test-table [id=304baf10cd574c4982d97f438ff3f845]) successfully deleted
I20260928 12:58:20.716307 13643 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: No bootstrap required, opened a new log
I20260928 12:58:20.716349 13643 ts_tablet_manager.cc:1397] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:20.716467 13643 raft_consensus.cc:348] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.716524 13643 raft_consensus.cc:374] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.716538 13643 raft_consensus.cc:729] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875b43f25de4f15a4394b8f43e59103, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.716589 13643 consensus_queue.cc:260] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.716711 13643 ts_tablet_manager.cc:1428] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:20.716711 13644 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Bootstrap starting.
I20260928 12:58:20.716775 13029 catalog_manager.cc:4841] TS 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631): tablet da2eb7aaa1204eb291560a9d5aa49f0d (table test-table [id=304baf10cd574c4982d97f438ff3f845]) successfully deleted
I20260928 12:58:20.717020 13644 tablet_bootstrap.cc:654] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:20.717187 13648 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e: No bootstrap required, opened a new log
I20260928 12:58:20.717245 13648 ts_tablet_manager.cc:1397] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:20.717350 13648 raft_consensus.cc:348] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.717437 13648 raft_consensus.cc:374] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.717466 13648 raft_consensus.cc:729] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.717525 13648 consensus_queue.cc:260] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.717595 13644 tablet_bootstrap.cc:492] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: No bootstrap required, opened a new log
I20260928 12:58:20.717634 13648 ts_tablet_manager.cc:1428] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:20.717643 13644 ts_tablet_manager.cc:1397] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:20.717753 13644 raft_consensus.cc:348] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.717833 13644 raft_consensus.cc:374] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:20.717859 13644 raft_consensus.cc:729] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d483a68d204cf8b1a9af1c2152af16, State: Initialized, Role: FOLLOWER
I20260928 12:58:20.717901 13644 consensus_queue.cc:260] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.717994 13644 ts_tablet_manager.cc:1428] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:20.786223 13602 raft_consensus.cc:482] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:20.786337 13602 raft_consensus.cc:504] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.786499 13602 leader_election.cc:288] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:20.786798 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf80b629e59942e4a08b1e072e0529e1" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103" is_pre_election: true
I20260928 12:58:20.786805 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf80b629e59942e4a08b1e072e0529e1" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16" is_pre_election: true
I20260928 12:58:20.786911 13463 raft_consensus.cc:2454] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:20.786911 13340 raft_consensus.cc:2454] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 0.
I20260928 12:58:20.787138 13132 leader_election.cc:302] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d483a68d204cf8b1a9af1c2152af16, 2f4dbca803e24703bcc793ff1874237e; no voters:
I20260928 12:58:20.787277 13602 raft_consensus.cc:2781] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:20.787329 13602 raft_consensus.cc:482] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:20.787353 13602 raft_consensus.cc:3037] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.788264 13602 raft_consensus.cc:504] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.788431 13602 leader_election.cc:288] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Requested vote from peers 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779), 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687)
I20260928 12:58:20.788614 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf80b629e59942e4a08b1e072e0529e1" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875b43f25de4f15a4394b8f43e59103"
I20260928 12:58:20.788710 13340 raft_consensus.cc:3037] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.788730 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf80b629e59942e4a08b1e072e0529e1" candidate_uuid: "2f4dbca803e24703bcc793ff1874237e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d483a68d204cf8b1a9af1c2152af16"
I20260928 12:58:20.788820 13463 raft_consensus.cc:3037] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:20.789558 13340 raft_consensus.cc:2454] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:20.789654 13463 raft_consensus.cc:2454] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4dbca803e24703bcc793ff1874237e in term 1.
I20260928 12:58:20.789821 13131 leader_election.cc:302] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4dbca803e24703bcc793ff1874237e, 3875b43f25de4f15a4394b8f43e59103; no voters:
I20260928 12:58:20.789916 13602 raft_consensus.cc:2781] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:20.789980 13602 raft_consensus.cc:686] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [term 1 LEADER]: Becoming Leader. State: Replica: 2f4dbca803e24703bcc793ff1874237e, State: Running, Role: LEADER
I20260928 12:58:20.790064 13602 consensus_queue.cc:237] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } }
I20260928 12:58:20.790691 13042 catalog_manager.cc:5495] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4dbca803e24703bcc793ff1874237e (127.11.11.65). New cstate: current_term: 1 leader_uuid: "2f4dbca803e24703bcc793ff1874237e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4dbca803e24703bcc793ff1874237e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:20.814575 13463 raft_consensus.cc:1261] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:20.814615 13340 raft_consensus.cc:1261] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Refusing update from remote peer 2f4dbca803e24703bcc793ff1874237e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:20.814837 13623 consensus_queue.cc:1035] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d483a68d204cf8b1a9af1c2152af16" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:20.814976 13602 consensus_queue.cc:1035] T cf80b629e59942e4a08b1e072e0529e1 P 2f4dbca803e24703bcc793ff1874237e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3875b43f25de4f15a4394b8f43e59103" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:58:20.819271 13615 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:58:22.339421 13042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:22.339525 13042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:22.340315 13042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 345387211ca2432cbe65904c73a3a6f2: Sending DeleteTablet for 3 replicas of tablet cf80b629e59942e4a08b1e072e0529e1
I20260928 12:58:22.340583 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet cf80b629e59942e4a08b1e072e0529e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:22 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53284
I20260928 12:58:22.340585 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet cf80b629e59942e4a08b1e072e0529e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:22 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59432
I20260928 12:58:22.340721 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13115
I20260928 12:58:22.340780 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet cf80b629e59942e4a08b1e072e0529e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:22 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:35492
I20260928 12:58:22.342893 13685 tablet_replica.cc:330] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: stopping tablet replica
I20260928 12:58:22.342954 13685 raft_consensus.cc:2229] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:22.342985 13685 raft_consensus.cc:2258] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:22.343230 13685 ts_tablet_manager.cc:1905] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:22.344177 13684 tablet_replica.cc:330] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: stopping tablet replica
I20260928 12:58:22.344272 13684 raft_consensus.cc:2229] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:22.344323 13684 raft_consensus.cc:2258] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:22.344559 13684 ts_tablet_manager.cc:1905] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:22.345494 13685 ts_tablet_manager.cc:1918] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:58:22.345566 13685 log.cc:1196] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/cf80b629e59942e4a08b1e072e0529e1
I20260928 12:58:22.345795 13685 ts_tablet_manager.cc:1939] T cf80b629e59942e4a08b1e072e0529e1 P 04d483a68d204cf8b1a9af1c2152af16: Deleting consensus metadata
I20260928 12:58:22.346169 13684 ts_tablet_manager.cc:1918] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:58:22.346246 13684 log.cc:1196] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/cf80b629e59942e4a08b1e072e0529e1
I20260928 12:58:22.346274 13028 catalog_manager.cc:4841] TS 04d483a68d204cf8b1a9af1c2152af16 (127.11.11.67:35687): tablet cf80b629e59942e4a08b1e072e0529e1 (table test-table [id=5804b9a2a6d2435f9736b9ac2c7ee63b]) successfully deleted
I20260928 12:58:22.346465 13684 ts_tablet_manager.cc:1939] T cf80b629e59942e4a08b1e072e0529e1 P 3875b43f25de4f15a4394b8f43e59103: Deleting consensus metadata
I20260928 12:58:22.346923 13027 catalog_manager.cc:4841] TS 3875b43f25de4f15a4394b8f43e59103 (127.11.11.66:36779): tablet cf80b629e59942e4a08b1e072e0529e1 (table test-table [id=5804b9a2a6d2435f9736b9ac2c7ee63b]) successfully deleted
W20260928 12:58:22.348466 13029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:38631 (error 108)
I20260928 12:58:22.348748 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13255
I20260928 12:58:22.350083 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:22.349868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38631 (local address 127.0.0.1:42514)
0928 12:58:22.349992 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:22.350049 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
W20260928 12:58:22.350231 13029 catalog_manager.cc:4584] TS 2f4dbca803e24703bcc793ff1874237e (127.11.11.65:38631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf80b629e59942e4a08b1e072e0529e1: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38631: connect: Connection refused (error 111)
I20260928 12:58:22.354979 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13395
I20260928 12:58:22.360417 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13011
2026-09-28T12:58:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5671 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:22 dist-test-slave-8sjz krb5kdc[13690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:22 dist-test-slave-8sjz krb5kdc[13690](info): set up 2 sockets
Sep 28 12:58:22 dist-test-slave-8sjz krb5kdc[13690](info): commencing operation
krb5kdc: starting...
W20260928 12:58:23.408883 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:23 dist-test-slave-8sjz krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:23.531198 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37793
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:37793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:23.614040 13703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:23.614266 13703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:23.614289 13703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:23.615397 13703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:23.615437 13703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:23.615451 13703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:23.615464 13703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:23.615476 13703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:23.617015 13703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:37793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:23.617233 13703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:23.617406 13703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:23.624451 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:23.624495 13709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:23.624451 13703 server_base.cc:1034] running on GCE node
W20260928 12:58:23.624539 13711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:23.624975 13703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:23.625298 13703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:23.626468 13703 hybrid_clock.cc:648] HybridClock initialized: now 1790600303626455 us; error 46 us; skew 500 ppm
Sep 28 12:58:23 dist-test-slave-8sjz krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:23.629127 13703 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:23.629585 13703 webserver.cc:460] Webserver started at http://127.11.11.126:38689/ using document root <none> and password file <none>
I20260928 12:58:23.629871 13703 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:23.629943 13703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:23.630090 13703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:23.631163 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "78d8bcd69ace44fc925427a87f84725a"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "dbd35ac8ab4aa6c92f46e4d5ccd85dc1"
server_key_iv: "a48d3171d5373107d41323efd74c8680"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.631528 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "78d8bcd69ace44fc925427a87f84725a"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "dbd35ac8ab4aa6c92f46e4d5ccd85dc1"
server_key_iv: "a48d3171d5373107d41323efd74c8680"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.633013 13703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:23.633713 13718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:23.633883 13703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:23.633987 13703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "78d8bcd69ace44fc925427a87f84725a"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "dbd35ac8ab4aa6c92f46e4d5ccd85dc1"
server_key_iv: "a48d3171d5373107d41323efd74c8680"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.634060 13703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:23.670753 13703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:23.672839 13703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:23.673020 13703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:23.677230 13703 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:37793
I20260928 12:58:23.677271 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:37793 every 8 connection(s)
I20260928 12:58:23.677642 13703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:23.678524 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:23.680861 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a: Bootstrap starting.
I20260928 12:58:23.681401 13771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:23.681633 13771 log.cc:824] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:23.682327 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a: No bootstrap required, opened a new log
I20260928 12:58:23.683911 13771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } }
I20260928 12:58:23.684124 13771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:23.684159 13771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d8bcd69ace44fc925427a87f84725a, State: Initialized, Role: FOLLOWER
I20260928 12:58:23.684314 13771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } }
I20260928 12:58:23.684397 13771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:23.684430 13771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:23.684465 13771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:23.684988 13771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } }
I20260928 12:58:23.685075 13771 leader_election.cc:302] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d8bcd69ace44fc925427a87f84725a; no voters:
I20260928 12:58:23.685202 13771 leader_election.cc:288] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:23.685261 13774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:23.685432 13774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [term 1 LEADER]: Becoming Leader. State: Replica: 78d8bcd69ace44fc925427a87f84725a, State: Running, Role: LEADER
I20260928 12:58:23.685487 13771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:23.685556 13774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } }
I20260928 12:58:23.685999 13776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d8bcd69ace44fc925427a87f84725a. Latest consensus state: current_term: 1 leader_uuid: "78d8bcd69ace44fc925427a87f84725a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } } }
I20260928 12:58:23.686085 13776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:23.686137 13775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d8bcd69ace44fc925427a87f84725a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d8bcd69ace44fc925427a87f84725a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37793 } } }
I20260928 12:58:23.686233 13775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:23.686501 13783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:23.686808 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13703
I20260928 12:58:23.686893 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:23.687067 11309 external_mini_cluster.cc:1428] Setting key f1f970e281608ce3056cceffe6f277eb
I20260928 12:58:23.687837 13783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:23.689611 13783 catalog_manager.cc:1347] Generated new cluster ID: fdd5670c854a4b369ff693a2cbfcd3fb
I20260928 12:58:23.689664 13783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:23 dist-test-slave-8sjz krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:23.704591 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:23.687694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.11.11.126:37793)
0928 12:58:23.688104 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:23.688114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:23.688146 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:58:23.688778 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:23.688790 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:23.688868 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:23.690262 (+ 1394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:23.690364 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.692761 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.694193 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.694855 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.695649 (+ 794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:23.698648 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:23.698653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:23.698653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:23.698666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:23.702133 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:23.702867 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:23.702871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:23.702872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:23.703021 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:23.703241 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:23.703242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:23.703243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:23.703861 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:23.704056 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:23.704178 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:58:23.704350 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":76,"threads_started":1}
I20260928 12:58:23.717423 13783 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:23.718350 13783 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:23.740257 13783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d8bcd69ace44fc925427a87f84725a: Generated new TSK 0
I20260928 12:58:23.740501 13783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:23.771865 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:23.868021 13797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:23.868299 13797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:23.868340 13797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:23.869551 13797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:23.869634 13797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:23.869675 13797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:23.871210 13797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:23.871505 13797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:23.871757 13797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:23.876853 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:23.876856 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:23.877147 13797 server_base.cc:1034] running on GCE node
W20260928 12:58:23.876873 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:23.877406 13797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:23.877610 13797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:23.878774 13797 hybrid_clock.cc:648] HybridClock initialized: now 1790600303878771 us; error 36 us; skew 500 ppm
Sep 28 12:58:23 dist-test-slave-8sjz krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:23.881319 13797 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:23.881955 13797 webserver.cc:460] Webserver started at http://127.11.11.65:40925/ using document root <none> and password file <none>
I20260928 12:58:23.882266 13797 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:23.882326 13797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:23.882463 13797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:23.883757 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "a6eaf340b65443419bd14a22b6d7a624"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "4d0a70ed3738114b74ae763e2706ba20"
server_key_iv: "ba710570d89593af0452fbdb3f1e29b1"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.884231 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "a6eaf340b65443419bd14a22b6d7a624"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "4d0a70ed3738114b74ae763e2706ba20"
server_key_iv: "ba710570d89593af0452fbdb3f1e29b1"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.885772 13797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:23.886629 13812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:23.886797 13797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:23.886898 13797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "a6eaf340b65443419bd14a22b6d7a624"
format_stamp: "Formatted at 2026-09-28 12:58:23 on dist-test-slave-8sjz"
server_key: "4d0a70ed3738114b74ae763e2706ba20"
server_key_iv: "ba710570d89593af0452fbdb3f1e29b1"
server_key_version: "encryptionkey@0"
I20260928 12:58:23.886987 13797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:23.897099 13797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:23.899478 13797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:23.899653 13797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:23.899902 13797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:23.900286 13797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:23.900342 13797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:23.900377 13797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:23.900404 13797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:23.906183 13797 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:37909
I20260928 12:58:23.906265 13925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:37909 every 8 connection(s)
I20260928 12:58:23.906613 13797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:23.907554 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13797
I20260928 12:58:23.907650 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:23.907838 11309 external_mini_cluster.cc:1428] Setting key 67205ac71d123b615e845c140d2c900a
Sep 28 12:58:23 dist-test-slave-8sjz krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 12:58:23.916824 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:23.907372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:57393 (local address 127.11.11.126:37793)
0928 12:58:23.907436 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:23.907438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:23.907709 (+ 271us) server_negotiation.cc:407] Connection header received
0928 12:58:23.908344 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:23.908345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:23.908362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:23.908409 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:23.909377 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.909893 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.910925 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.911141 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.911228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:23.913801 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:23.913803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:23.913803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:23.913811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:23.915264 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:23.915718 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:23.915719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:23.915720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:23.915780 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:23.916065 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:23.916066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:23.916066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:23.916189 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:23.916299 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:23.916570 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 12:58:23.916728 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:58:23.916970 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:23.907339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37793 (local address 127.11.11.65:57393)
0928 12:58:23.907520 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:23.907556 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:23.908230 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:23.908480 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:23.908485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:23.909068 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:23.909272 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:23.909277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.909999 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.910000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:23.910821 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:23.910823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:23.911240 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:23.911240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:23.911371 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:23.911944 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:23.911952 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:23.913694 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:23.915368 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:23.915375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:23.915378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:23.915628 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:23.915849 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:23.915850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:23.915851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:23.915969 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:23.916265 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:23.916272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:23.916432 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 12:58:23.916548 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 12:58:23.916732 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 12:58:23.917487 13926 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37793
I20260928 12:58:23.917620 13926 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:23.917846 13926 heartbeater.cc:511] Master 127.11.11.126:37793 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
I20260928 12:58:23.918499 13735 ts_manager.cc:194] Registered new tserver with Master: a6eaf340b65443419bd14a22b6d7a624 (127.11.11.65:37909)
I20260928 12:58:23.919445 13735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:57393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:23.935606 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:24.022327 13931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:24.022552 13931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:24.022578 13931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:24.023834 13931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:24.023888 13931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:24.023918 13931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:24.025585 13931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:24.025826 13931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:24.026022 13931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:24.031605 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:24.031569 13939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:24.031569 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:24.031929 13931 server_base.cc:1034] running on GCE node
I20260928 12:58:24.032217 13931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:24.032547 13931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:24.033758 13931 hybrid_clock.cc:648] HybridClock initialized: now 1790600304033736 us; error 36 us; skew 500 ppm
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600304, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:24.036965 13931 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:24.037542 13931 webserver.cc:460] Webserver started at http://127.11.11.66:44445/ using document root <none> and password file <none>
I20260928 12:58:24.037813 13931 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:24.037863 13931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:24.038000 13931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:24.039116 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "90ae0a1b5a774dad9b3e2fe4ea61afb3"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "225a67b7d7133570b8e13e7277f1de5e"
server_key_iv: "108da7d5a24042dacffb7a3671e94736"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.039510 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "90ae0a1b5a774dad9b3e2fe4ea61afb3"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "225a67b7d7133570b8e13e7277f1de5e"
server_key_iv: "108da7d5a24042dacffb7a3671e94736"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.041043 13931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:24.041931 13946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.042172 13931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:24.042279 13931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "90ae0a1b5a774dad9b3e2fe4ea61afb3"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "225a67b7d7133570b8e13e7277f1de5e"
server_key_iv: "108da7d5a24042dacffb7a3671e94736"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.042342 13931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:24.062058 13931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:24.064576 13931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:24.064715 13931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:24.064913 13931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:24.065275 13931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:24.065313 13931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.065375 13931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:24.065413 13931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.071861 13931 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:46637
I20260928 12:58:24.071954 14059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:46637 every 8 connection(s)
I20260928 12:58:24.072305 13931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600304, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:24.081293 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 13931
I20260928 12:58:24.081418 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:24.081655 11309 external_mini_cluster.cc:1428] Setting key 08704d9dfd391f5a92cb14585ddbf474
I20260928 12:58:24.083958 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:24.073272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49207 (local address 127.11.11.126:37793)
0928 12:58:24.073351 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:24.073353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:24.073622 (+ 269us) server_negotiation.cc:407] Connection header received
0928 12:58:24.074153 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:24.074154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:24.074167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:24.074237 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:24.075659 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.076258 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.077393 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.077617 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.077696 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:24.080609 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:24.080611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:24.080611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:24.080619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:24.082362 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:24.082824 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:24.082825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:24.082826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:24.082897 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:24.083210 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:24.083211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:24.083211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:24.083368 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:24.083442 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:24.083728 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 12:58:24.083834 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:58:24.084445 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:24.073258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37793 (local address 127.11.11.66:49207)
0928 12:58:24.073507 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:24.073541 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:24.074018 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:24.074397 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:24.074402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:24.075248 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:24.075536 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:24.075542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.076356 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.076357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:24.077284 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:24.077287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.077707 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.077708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:24.077833 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:24.078530 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:24.078541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:24.080487 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:24.082463 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:24.082478 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:24.082481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:24.082736 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:24.082970 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:24.082970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:24.082971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:24.083134 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:24.083434 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:24.083438 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:24.083582 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 12:58:24.083703 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 12:58:24.083950 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260928 12:58:24.085059 14060 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37793
I20260928 12:58:24.085186 14060 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:24.085426 14060 heartbeater.cc:511] Master 127.11.11.126:37793 requested a full tablet report, sending...
I20260928 12:58:24.085834 13735 ts_manager.cc:194] Registered new tserver with Master: 90ae0a1b5a774dad9b3e2fe4ea61afb3 (127.11.11.66:46637)
I20260928 12:58:24.086302 13735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:49207
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
W20260928 12:58:24.275453 11309 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.11.67: real 0.193s user 0.000s sys 0.003s
I20260928 12:58:24.275885 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:24.362787 14065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:24.363018 14065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:24.363044 14065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:24.364432 14065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:24.364490 14065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:24.364521 14065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:24.366129 14065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:24.366429 14065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:24.366690 14065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:24.371882 14071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:24.371925 14070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:24.372076 14073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:24.372136 14065 server_base.cc:1034] running on GCE node
I20260928 12:58:24.372351 14065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:24.372622 14065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:24.373780 14065 hybrid_clock.cc:648] HybridClock initialized: now 1790600304373765 us; error 31 us; skew 500 ppm
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600304, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:24.376322 14065 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:24.376783 14065 webserver.cc:460] Webserver started at http://127.11.11.67:33995/ using document root <none> and password file <none>
I20260928 12:58:24.376993 14065 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:24.377034 14065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:24.377127 14065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:24.378166 14065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "c6decd3d5ecd42ccb1e2ab42a5f337fd"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "ce18ffb7410704d5a4590072dc727b4b"
server_key_iv: "08b236b5b6760713c8547166a11700f3"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.378465 14065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "c6decd3d5ecd42ccb1e2ab42a5f337fd"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "ce18ffb7410704d5a4590072dc727b4b"
server_key_iv: "08b236b5b6760713c8547166a11700f3"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.379778 14065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:58:24.380649 14080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.380971 14065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:24.381073 14065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "c6decd3d5ecd42ccb1e2ab42a5f337fd"
format_stamp: "Formatted at 2026-09-28 12:58:24 on dist-test-slave-8sjz"
server_key: "ce18ffb7410704d5a4590072dc727b4b"
server_key_iv: "08b236b5b6760713c8547166a11700f3"
server_key_version: "encryptionkey@0"
I20260928 12:58:24.381155 14065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:24.394502 14065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:24.396744 14065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:24.396869 14065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:24.397063 14065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:24.397392 14065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:24.397428 14065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.397475 14065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:24.397500 14065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:24.403613 14065 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:36175
I20260928 12:58:24.403649 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:36175 every 8 connection(s)
I20260928 12:58:24.404009 14065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600304, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:24.411245 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14065
I20260928 12:58:24.411370 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:24.411569 11309 external_mini_cluster.cc:1428] Setting key e432d59d6b2d2eff8e732a58f6585161
I20260928 12:58:24.414215 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:24.404865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:48271 (local address 127.11.11.126:37793)
0928 12:58:24.404931 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:24.404933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:24.405126 (+ 193us) server_negotiation.cc:407] Connection header received
0928 12:58:24.405618 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:24.405619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:24.405629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:24.405696 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:24.406674 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.407228 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.408416 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.408608 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.408691 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:24.411073 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:24.411074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:24.411075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:24.411082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:24.412923 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:24.413310 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:24.413310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:24.413311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:24.413361 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:24.413575 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:24.413575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:24.413576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:24.413677 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:24.413727 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:24.414075 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 12:58:24.414139 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:58:24.414532 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:24.404815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37793 (local address 127.11.11.67:48271)
0928 12:58:24.404982 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:24.405027 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:24.405534 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:24.405768 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:24.405771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:24.406360 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:24.406565 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:24.406569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.407361 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.407362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:24.408329 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:24.408332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:24.408705 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:24.408706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:24.408824 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:24.409370 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:24.409379 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:24.410972 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:24.412998 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:24.413005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:24.413007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:24.413246 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:24.413419 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:24.413420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:24.413420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:24.413522 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:24.413795 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:24.413801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:24.413942 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:58:24.414054 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:58:24.414202 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260928 12:58:24.415138 14194 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37793
I20260928 12:58:24.415263 14194 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:24.415465 14194 heartbeater.cc:511] Master 127.11.11.126:37793 requested a full tablet report, sending...
I20260928 12:58:24.415910 13735 ts_manager.cc:194] Registered new tserver with Master: c6decd3d5ecd42ccb1e2ab42a5f337fd (127.11.11.67:36175)
I20260928 12:58:24.416325 13735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:48271
I20260928 12:58:24.423772 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:58:24.435904 13793 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:24.434862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.11.11.126:37793)
0928 12:58:24.434958 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:24.434961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:24.435058 (+ 97us) server_negotiation.cc:407] Connection header received
0928 12:58:24.435159 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:24.435160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:24.435171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:24.435246 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:24.435843 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53360: BlockingRecv error: recv got EOF from 127.0.0.1:53360 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 12:58:24.437067 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13797
I20260928 12:58:24.441833 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13931
I20260928 12:58:24.446689 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14065
I20260928 12:58:24.451085 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 13703
2026-09-28T12:58:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2089 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[14206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[14206](info): set up 2 sockets
Sep 28 12:58:24 dist-test-slave-8sjz krb5kdc[14206](info): commencing operation
krb5kdc: starting...
W20260928 12:58:25.492974 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:25.620205 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:33017
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:25.703807 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:25.704092 14219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:25.704154 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:25.705439 14219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:25.705524 14219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:25.705541 14219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:25.705606 14219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:25.705626 14219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:25.707294 14219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:33017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:25.707580 14219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:25.707834 14219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:25.712925 14225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:25.712955 14224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:25.712925 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:25.713251 14219 server_base.cc:1034] running on GCE node
I20260928 12:58:25.713521 14219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:25.713846 14219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:25.715044 14219 hybrid_clock.cc:648] HybridClock initialized: now 1790600305715022 us; error 45 us; skew 500 ppm
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600305, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:25.723737 14219 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:25.724318 14219 webserver.cc:460] Webserver started at http://127.11.11.126:38041/ using document root <none> and password file <none>
I20260928 12:58:25.724567 14219 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:25.724617 14219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:25.724798 14219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:25.725945 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "73dd86c5fc7e485ba6300289f5b94f82"
format_stamp: "Formatted at 2026-09-28 12:58:25 on dist-test-slave-8sjz"
server_key: "baaf89b3bf1c385399530b5cbfaaa23f"
server_key_iv: "04895852777161f289712284a6698409"
server_key_version: "encryptionkey@0"
I20260928 12:58:25.726281 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "73dd86c5fc7e485ba6300289f5b94f82"
format_stamp: "Formatted at 2026-09-28 12:58:25 on dist-test-slave-8sjz"
server_key: "baaf89b3bf1c385399530b5cbfaaa23f"
server_key_iv: "04895852777161f289712284a6698409"
server_key_version: "encryptionkey@0"
I20260928 12:58:25.727715 14219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:25.728533 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:25.728734 14219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:25.728833 14219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "73dd86c5fc7e485ba6300289f5b94f82"
format_stamp: "Formatted at 2026-09-28 12:58:25 on dist-test-slave-8sjz"
server_key: "baaf89b3bf1c385399530b5cbfaaa23f"
server_key_iv: "04895852777161f289712284a6698409"
server_key_version: "encryptionkey@0"
I20260928 12:58:25.728899 14219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:25.743391 14219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:25.745968 14219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:25.746097 14219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:25.750198 14219 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:33017
I20260928 12:58:25.750274 14286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:33017 every 8 connection(s)
I20260928 12:58:25.750788 14219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:25.751477 14287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:25.754004 14287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82: Bootstrap starting.
I20260928 12:58:25.754642 14287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:25.754957 14287 log.cc:824] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:25.755326 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14219
I20260928 12:58:25.755439 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:25.755641 11309 external_mini_cluster.cc:1428] Setting key 9085a39995361279b379217695808815
I20260928 12:58:25.755914 14287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82: No bootstrap required, opened a new log
I20260928 12:58:25.758272 14287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } }
I20260928 12:58:25.758455 14287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:25.758520 14287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73dd86c5fc7e485ba6300289f5b94f82, State: Initialized, Role: FOLLOWER
I20260928 12:58:25.758700 14287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } }
I20260928 12:58:25.758803 14287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:25.758850 14287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:25.758906 14287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:25.759509 14287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } }
I20260928 12:58:25.759634 14287 leader_election.cc:302] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73dd86c5fc7e485ba6300289f5b94f82; no voters:
I20260928 12:58:25.759843 14287 leader_election.cc:288] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:25.759873 14292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:25.759994 14292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [term 1 LEADER]: Becoming Leader. State: Replica: 73dd86c5fc7e485ba6300289f5b94f82, State: Running, Role: LEADER
I20260928 12:58:25.760188 14287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:25.760160 14292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } }
I20260928 12:58:25.760743 14293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73dd86c5fc7e485ba6300289f5b94f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } } }
I20260928 12:58:25.760841 14293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:25.761075 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73dd86c5fc7e485ba6300289f5b94f82. Latest consensus state: current_term: 1 leader_uuid: "73dd86c5fc7e485ba6300289f5b94f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73dd86c5fc7e485ba6300289f5b94f82" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33017 } } }
I20260928 12:58:25.761365 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:25.761586 14299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:25.762142 14299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:25.765338 14299 catalog_manager.cc:1347] Generated new cluster ID: 0b5fac74fbb645acb0adf0cb67109ef4
I20260928 12:58:25.765422 14299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:25.768925 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:25.756301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.11.11.126:33017)
0928 12:58:25.756508 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:25.756516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:25.756544 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:58:25.758251 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:25.758260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:25.758323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:25.758497 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:25.759265 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:25.760363 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:25.761464 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:25.762024 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:25.762170 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:25.765295 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:25.765299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:25.765299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:25.765310 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:25.767161 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:25.767591 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:25.767594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:25.767595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:25.767713 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:25.767939 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:25.767940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:25.767941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:25.768309 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:25.768498 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:25.768581 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:58:25.768740 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 12:58:25.769995 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:58:25.776141 14299 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:25.776736 14299 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:25.782637 14299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73dd86c5fc7e485ba6300289f5b94f82: Generated new TSK 0
I20260928 12:58:25.782853 14299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:25.789616 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:25.785612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.11.11.126:33017)
0928 12:58:25.785667 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:25.785669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:25.785801 (+ 132us) server_negotiation.cc:407] Connection header received
0928 12:58:25.785871 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:25.785872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:25.785881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:25.785943 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:25.786403 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:25.786907 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:25.788083 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:25.788311 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:25.788371 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:25.789129 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:25.789130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:25.789131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:25.789360 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:25.789376 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:25.789382 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:25.789430 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:25.789527 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 12:58:25.789554 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:58:25.792466 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14219
2026-09-28T12:58:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1339 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14320](info): set up 2 sockets
Sep 28 12:58:25 dist-test-slave-8sjz krb5kdc[14320](info): commencing operation
krb5kdc: starting...
W20260928 12:58:26.876843 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:26 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:27.003489 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35031
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:27.090080 14333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:27.090327 14333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:27.090351 14333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:27.091524 14333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:27.091570 14333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:27.091585 14333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:27.091598 14333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:27.091610 14333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:27.093312 14333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:35031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:27.093554 14333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:27.093757 14333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:58:27.099018 14333 server_base.cc:1034] running on GCE node
W20260928 12:58:27.100337 14339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.100533 14341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.101454 14338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:27.101784 14333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:27.102097 14333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:27.103260 14333 hybrid_clock.cc:648] HybridClock initialized: now 1790600307103237 us; error 31 us; skew 500 ppm
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:27.106187 14333 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:27.106683 14333 webserver.cc:460] Webserver started at http://127.11.11.126:46191/ using document root <none> and password file <none>
I20260928 12:58:27.106922 14333 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:27.106992 14333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:27.107157 14333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:27.108376 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "6f2605fc812346dc9d8cad52899a02ee"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4a97deadaf31cd589f6cb0fb8a0d7aed"
server_key_iv: "23ef081633d74b19c165fa2e96873c39"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.108788 14333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "6f2605fc812346dc9d8cad52899a02ee"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4a97deadaf31cd589f6cb0fb8a0d7aed"
server_key_iv: "23ef081633d74b19c165fa2e96873c39"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.110294 14333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:58:27.111162 14348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.111374 14333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:27.111483 14333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "6f2605fc812346dc9d8cad52899a02ee"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4a97deadaf31cd589f6cb0fb8a0d7aed"
server_key_iv: "23ef081633d74b19c165fa2e96873c39"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.111562 14333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:27.137283 14333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:27.139840 14333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:27.140025 14333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:27.144129 14333 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:35031
I20260928 12:58:27.144165 14400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:35031 every 8 connection(s)
I20260928 12:58:27.144578 14333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:27.145696 14401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:27.148233 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: Bootstrap starting.
I20260928 12:58:27.148904 14401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:27.149047 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14333
I20260928 12:58:27.149164 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:27.149281 14401 log.cc:824] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:27.149380 11309 external_mini_cluster.cc:1428] Setting key 60bdf487851be772b5469ad1a02750c7
I20260928 12:58:27.150122 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: No bootstrap required, opened a new log
I20260928 12:58:27.152446 14401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } }
I20260928 12:58:27.152621 14401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:27.152669 14401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2605fc812346dc9d8cad52899a02ee, State: Initialized, Role: FOLLOWER
I20260928 12:58:27.152801 14401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } }
I20260928 12:58:27.152918 14401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:27.152962 14401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:27.153012 14401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:27.153647 14401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } }
I20260928 12:58:27.153779 14401 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2605fc812346dc9d8cad52899a02ee; no voters:
I20260928 12:58:27.153967 14401 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:27.154016 14406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:27.154179 14406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [term 1 LEADER]: Becoming Leader. State: Replica: 6f2605fc812346dc9d8cad52899a02ee, State: Running, Role: LEADER
I20260928 12:58:27.154258 14401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:27.154981 14406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } }
I20260928 12:58:27.155592 14408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2605fc812346dc9d8cad52899a02ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } } }
I20260928 12:58:27.155678 14408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:27.155853 14407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2605fc812346dc9d8cad52899a02ee. Latest consensus state: current_term: 1 leader_uuid: "6f2605fc812346dc9d8cad52899a02ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2605fc812346dc9d8cad52899a02ee" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35031 } } }
I20260928 12:58:27.156118 14407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee [sys.catalog]: This master's current role is: LEADER
W20260928 12:58:27.156270 14420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:27.156356 14420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:27.156442 14421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:27.156628 14421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:27.158588 14421 catalog_manager.cc:1347] Generated new cluster ID: f2b7636554954504b28ac21c8b337e68
I20260928 12:58:27.158641 14421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:27.160856 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.149987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.11.11.126:35031)
0928 12:58:27.150193 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:27.150204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:27.150238 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:58:27.150677 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:27.150683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:27.150734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:27.150905 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:27.151559 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.152511 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.153431 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.153940 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.154067 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.157233 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:27.157237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:27.157238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:27.157251 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:27.159180 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.159573 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.159576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.159577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.159678 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.159893 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.159894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.159894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.160240 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:27.160419 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:27.160525 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:58:27.160673 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260928 12:58:27.174039 14421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:27.174644 14421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:27.179286 14421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2605fc812346dc9d8cad52899a02ee: Generated new TSK 0
I20260928 12:58:27.179466 14421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:27.215090 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:27.325446 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:27.325744 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:27.325793 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:27.327119 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:27.327193 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:27.327245 14427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:27.328951 14427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:27.329269 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:27.329533 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:27.335186 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.335192 14433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.335189 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:27.335510 14427 server_base.cc:1034] running on GCE node
I20260928 12:58:27.335785 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:27.336105 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:27.337309 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790600307337283 us; error 34 us; skew 500 ppm
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:27.339974 14427 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:27.340516 14427 webserver.cc:460] Webserver started at http://127.11.11.65:34717/ using document root <none> and password file <none>
I20260928 12:58:27.340807 14427 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:27.340893 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:27.341054 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:27.342119 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "bdacc5c5b4cc4053a1b81761d9ed7a3f"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "f111d7c5a3943acbb764fc293c5e6e29"
server_key_iv: "e23b66aac71e05aeb6c045d730180a25"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.342501 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "bdacc5c5b4cc4053a1b81761d9ed7a3f"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "f111d7c5a3943acbb764fc293c5e6e29"
server_key_iv: "e23b66aac71e05aeb6c045d730180a25"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.343961 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 12:58:27.344987 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.345171 14427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:27.345273 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "bdacc5c5b4cc4053a1b81761d9ed7a3f"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "f111d7c5a3943acbb764fc293c5e6e29"
server_key_iv: "e23b66aac71e05aeb6c045d730180a25"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.345362 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:27.366694 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:27.369278 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:27.369462 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:27.369750 14427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:27.370121 14427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:27.370182 14427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.370220 14427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:27.370250 14427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.376695 14427 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:36447
I20260928 12:58:27.376746 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:36447 every 8 connection(s)
I20260928 12:58:27.377128 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:27.381541 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14427
I20260928 12:58:27.381647 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:27.381847 11309 external_mini_cluster.cc:1428] Setting key db3bfdef89be10e19d4ed60316744403
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:27.387910 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.377973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45217 (local address 127.11.11.126:35031)
0928 12:58:27.378025 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:27.378028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:27.378266 (+ 238us) server_negotiation.cc:407] Connection header received
0928 12:58:27.378789 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:27.378790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:27.378802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:27.378867 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:27.379852 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.380450 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.381535 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.381842 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.381953 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.384510 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:27.384512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:27.384512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:27.384522 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:27.386321 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.386824 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.386826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.386827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.386900 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.387149 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.387150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.387150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.387320 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:27.387398 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:27.387662 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 12:58:27.387763 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:27.387934 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.377967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35031 (local address 127.11.11.65:45217)
0928 12:58:27.378134 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:27.378172 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:27.378664 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:27.378941 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:27.378944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:27.379545 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:27.379749 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.379754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.380539 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.380539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.381440 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.381443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.381964 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.381965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.382108 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.382696 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:27.382705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:27.384374 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:27.386413 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.386421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.386423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.386738 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.386971 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.386972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.386972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.387079 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.387389 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:27.387393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:27.387521 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 12:58:27.387609 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:58:27.387741 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 12:58:27.388656 14556 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35031
I20260928 12:58:27.388784 14556 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:27.389011 14556 heartbeater.cc:511] Master 127.11.11.126:35031 requested a full tablet report, sending...
I20260928 12:58:27.389676 14365 ts_manager.cc:194] Registered new tserver with Master: bdacc5c5b4cc4053a1b81761d9ed7a3f (127.11.11.65:36447)
I20260928 12:58:27.390750 14365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:45217
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:27.411551 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:27.495882 14561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:27.496137 14561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:27.496165 14561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:27.497427 14561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:27.497479 14561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:27.497509 14561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:27.499101 14561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:27.499392 14561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:27.499641 14561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:27.504859 14566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.504879 14567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.504868 14569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:27.505173 14561 server_base.cc:1034] running on GCE node
I20260928 12:58:27.505424 14561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:27.505705 14561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:27.506917 14561 hybrid_clock.cc:648] HybridClock initialized: now 1790600307506903 us; error 32 us; skew 500 ppm
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:27.509820 14561 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:27.510293 14561 webserver.cc:460] Webserver started at http://127.11.11.66:39069/ using document root <none> and password file <none>
I20260928 12:58:27.510529 14561 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:27.510607 14561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:27.510763 14561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:27.511855 14561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "42526aac07be48379ca8d0573db6cc2c"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4563ee0de7003509204721352bb5b876"
server_key_iv: "50721487c4750c128b4d0f026227c0f8"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.512260 14561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "42526aac07be48379ca8d0573db6cc2c"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4563ee0de7003509204721352bb5b876"
server_key_iv: "50721487c4750c128b4d0f026227c0f8"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.513737 14561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:27.514604 14576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.514981 14561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:27.515098 14561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "42526aac07be48379ca8d0573db6cc2c"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "4563ee0de7003509204721352bb5b876"
server_key_iv: "50721487c4750c128b4d0f026227c0f8"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.515173 14561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:27.533242 14561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:27.535641 14561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:27.535791 14561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:27.536113 14561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:27.536470 14561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:27.536526 14561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.536563 14561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:27.536590 14561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.543148 14561 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:33247
I20260928 12:58:27.543186 14689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:33247 every 8 connection(s)
I20260928 12:58:27.543526 14561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:27.547333 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14561
I20260928 12:58:27.547444 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:27.547638 11309 external_mini_cluster.cc:1428] Setting key 6f49c427cd2a1f230a6d0b1f019f925c
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:27.554467 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.544428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:35449 (local address 127.11.11.126:35031)
0928 12:58:27.544512 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:27.544515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:27.544644 (+ 129us) server_negotiation.cc:407] Connection header received
0928 12:58:27.545279 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:27.545281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:27.545292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:27.545346 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:27.546364 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.546896 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.548356 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.548647 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.548742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.551315 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:27.551316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:27.551316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:27.551324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:27.552837 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.553377 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.553377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.553378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.553429 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.553778 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.553779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.553779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.553888 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:27.553942 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:27.554332 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 12:58:27.554400 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:58:27.554742 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.544347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35031 (local address 127.11.11.66:35449)
0928 12:58:27.544510 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:27.544548 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:27.545168 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:27.545413 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:27.545417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:27.546041 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:27.546250 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.546255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.546984 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.546985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.548267 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.548270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.548755 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.548756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.548870 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.549517 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:27.549529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:27.551210 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:27.552923 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.552933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.552936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.553298 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.553498 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.553499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.553499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.553657 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.553956 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:27.553962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:27.554142 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 12:58:27.554284 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 12:58:27.554475 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 12:58:27.555265 14690 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35031
I20260928 12:58:27.555398 14690 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:27.555588 14690 heartbeater.cc:511] Master 127.11.11.126:35031 requested a full tablet report, sending...
I20260928 12:58:27.556066 14365 ts_manager.cc:194] Registered new tserver with Master: 42526aac07be48379ca8d0573db6cc2c (127.11.11.66:33247)
I20260928 12:58:27.556437 14365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:35449
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:27.576395 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:27.659929 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:27.660233 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:27.660269 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:27.661558 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:27.661605 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:27.661633 14695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:27.663080 14695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:27.663300 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:27.663496 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:27.668588 14701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:27.668699 14695 server_base.cc:1034] running on GCE node
W20260928 12:58:27.668588 14703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:27.668609 14700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:27.669001 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:27.669289 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:27.670462 14695 hybrid_clock.cc:648] HybridClock initialized: now 1790600307670441 us; error 31 us; skew 500 ppm
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:27.673274 14695 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:27.673708 14695 webserver.cc:460] Webserver started at http://127.11.11.67:37441/ using document root <none> and password file <none>
I20260928 12:58:27.673911 14695 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:27.673960 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:27.674053 14695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:27.675084 14695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "cab6f44037f941a8a9f62f3b0cf7caf6"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "cab2e6d95a9d3e5015c2038d9ed2c29c"
server_key_iv: "9615f54b8b5cabbb939bc22dd32d42ef"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.675392 14695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "cab6f44037f941a8a9f62f3b0cf7caf6"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "cab2e6d95a9d3e5015c2038d9ed2c29c"
server_key_iv: "9615f54b8b5cabbb939bc22dd32d42ef"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.676815 14695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:27.677619 14710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.677819 14695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:27.677938 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "cab6f44037f941a8a9f62f3b0cf7caf6"
format_stamp: "Formatted at 2026-09-28 12:58:27 on dist-test-slave-8sjz"
server_key: "cab2e6d95a9d3e5015c2038d9ed2c29c"
server_key_iv: "9615f54b8b5cabbb939bc22dd32d42ef"
server_key_version: "encryptionkey@0"
I20260928 12:58:27.678036 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:27.710232 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:27.712819 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:27.713035 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:27.713299 14695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:27.713634 14695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:27.713697 14695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.713766 14695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:27.713814 14695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:27.719923 14695 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:46133
I20260928 12:58:27.719966 14823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:46133 every 8 connection(s)
I20260928 12:58:27.720347 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:27.721818 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14695
I20260928 12:58:27.721923 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:27.722117 11309 external_mini_cluster.cc:1428] Setting key e098ccf370b7147a3fe829a7b4f8e8b6
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:27.729789 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.721138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:54685 (local address 127.11.11.126:35031)
0928 12:58:27.721210 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:27.721212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:27.721403 (+ 191us) server_negotiation.cc:407] Connection header received
0928 12:58:27.721896 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:27.721898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:27.721914 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:27.721992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:27.723246 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.723752 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.724806 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.724999 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.725067 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.727254 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:27.727255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:27.727256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:27.727262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:27.728536 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.728905 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.728905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.728906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.728955 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.729193 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.729194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.729195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.729323 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:27.729398 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:27.729639 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:58:27.729709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:58:27.729959 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.721098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35031 (local address 127.11.11.67:54685)
0928 12:58:27.721257 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:27.721315 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:27.721800 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:27.722093 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:27.722096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:27.722927 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:27.723164 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.723169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.723833 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.723833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.724712 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:27.724714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.725090 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.725090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:27.725202 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.725790 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:27.725800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:27.727167 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:27.728607 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.728613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.728615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.728840 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.729011 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:27.729012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:27.729012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:27.729110 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:27.729396 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:27.729400 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:27.729522 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 12:58:27.729607 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 12:58:27.729730 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 12:58:27.730537 14824 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35031
I20260928 12:58:27.730649 14824 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:27.730845 14824 heartbeater.cc:511] Master 127.11.11.126:35031 requested a full tablet report, sending...
I20260928 12:58:27.731309 14365 ts_manager.cc:194] Registered new tserver with Master: cab6f44037f941a8a9f62f3b0cf7caf6 (127.11.11.67:46133)
I20260928 12:58:27.731745 14365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:54685
I20260928 12:58:27.733839 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:27.757833 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:27.749896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.11.11.126:35031)
0928 12:58:27.750014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:27.750016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:27.750123 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:58:27.750238 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:27.750239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:27.750251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:27.750322 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:27.750935 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.751437 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.752272 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:27.752474 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:27.752551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:27.755211 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:27.755213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:27.755213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:27.755221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:27.756749 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.757112 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.757113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.757113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.757162 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:27.757328 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:27.757329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:27.757329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:27.757475 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:27.757549 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:27.757664 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:58:27.757725 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 12:58:27.758112 14365 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58766
I20260928 12:58:27.758569 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14427
I20260928 12:58:27.763494 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14561
I20260928 12:58:27.767751 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14695
I20260928 12:58:27.773885 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14333
2026-09-28T12:58:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1983 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14836](info): set up 2 sockets
Sep 28 12:58:27 dist-test-slave-8sjz krb5kdc[14836](info): commencing operation
krb5kdc: starting...
W20260928 12:58:28.845085 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:28 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:28.971459 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45699
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:29.068318 14849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:29.068547 14849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:29.068572 14849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:29.069832 14849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:29.069878 14849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:29.069892 14849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:29.069905 14849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:29.069945 14849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:29.071568 14849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:29.071907 14849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:29.072182 14849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:29.077059 14854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:29.077059 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:29.077095 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:29.077392 14849 server_base.cc:1034] running on GCE node
I20260928 12:58:29.077651 14849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:29.077913 14849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:29.079087 14849 hybrid_clock.cc:648] HybridClock initialized: now 1790600309079074 us; error 32 us; skew 500 ppm
Sep 28 12:58:29 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600309, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:29.081892 14849 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:29.082315 14849 webserver.cc:460] Webserver started at http://127.11.11.126:32891/ using document root <none> and password file <none>
I20260928 12:58:29.082504 14849 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:29.082545 14849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:29.082639 14849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:29.083731 14849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "cf5d95a8da69ca4f67feab46f5f7e370"
server_key_iv: "d3add96a8500a7f4d4528deb69acf428"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.084110 14849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "cf5d95a8da69ca4f67feab46f5f7e370"
server_key_iv: "d3add96a8500a7f4d4528deb69acf428"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.085515 14849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:29.086437 14864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:29.086761 14849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:29.086853 14849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "cf5d95a8da69ca4f67feab46f5f7e370"
server_key_iv: "d3add96a8500a7f4d4528deb69acf428"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.086967 14849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:29.096154 14849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:29.098495 14849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:29.098624 14849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:29.102463 14849 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45699
I20260928 12:58:29.102543 14916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45699 every 8 connection(s)
I20260928 12:58:29.103071 14849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:29.103817 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:29.106324 14917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3: Bootstrap starting.
I20260928 12:58:29.106806 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14849
I20260928 12:58:29.106889 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:29.107048 14917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:29.107102 11309 external_mini_cluster.cc:1428] Setting key e577bf82f043e0654dd4816cdfddc95a
I20260928 12:58:29.107406 14917 log.cc:824] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:29.108292 14917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3: No bootstrap required, opened a new log
I20260928 12:58:29.110845 14917 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } }
I20260928 12:58:29.111032 14917 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:29.111078 14917 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f93d8e0fce43a9bbc717d10c2a25d3, State: Initialized, Role: FOLLOWER
I20260928 12:58:29.111248 14917 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } }
I20260928 12:58:29.111333 14917 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:29.111357 14917 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:29.111423 14917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:29.112135 14917 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } }
I20260928 12:58:29.112243 14917 leader_election.cc:302] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9f93d8e0fce43a9bbc717d10c2a25d3; no voters:
I20260928 12:58:29.112418 14917 leader_election.cc:288] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:29.112501 14922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:29.112673 14917 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:29.112684 14922 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [term 1 LEADER]: Becoming Leader. State: Replica: f9f93d8e0fce43a9bbc717d10c2a25d3, State: Running, Role: LEADER
I20260928 12:58:29.112849 14922 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } }
I20260928 12:58:29.113191 14924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9f93d8e0fce43a9bbc717d10c2a25d3. Latest consensus state: current_term: 1 leader_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } } }
I20260928 12:58:29.113276 14924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:29.113509 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9f93d8e0fce43a9bbc717d10c2a25d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45699 } } }
I20260928 12:58:29.113688 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:29.113517 14930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:29.114284 14930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:58:29 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:29.116016 14930 catalog_manager.cc:1347] Generated new cluster ID: ff22c1ed8cfc4a08a6cce48ead2142f6
I20260928 12:58:29.116097 14930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:29.119400 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:29.107697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.11.11.126:45699)
0928 12:58:29.107928 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:29.107938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:29.107974 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:58:29.108649 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:29.108659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:29.108732 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:29.108931 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:29.109543 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.110994 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.111902 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.112622 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.113072 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:29.115828 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:29.115832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:29.115832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:29.115845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:29.117758 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:29.118171 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:29.118175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:29.118176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:29.118280 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:29.118500 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:29.118501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:29.118501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:29.118777 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:29.118952 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:29.119056 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:58:29.119203 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 12:58:29.135542 14930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:29.136374 14930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:29.149781 14930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9f93d8e0fce43a9bbc717d10c2a25d3: Generated new TSK 0
I20260928 12:58:29.150058 14930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:29.191102 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:29.294538 14943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:29.294813 14943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:29.294859 14943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:29.296132 14943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:29.296202 14943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:29.296248 14943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:29.297907 14943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:29.298211 14943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:29.298465 14943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:29.303676 14951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:29.303669 14948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:29.303671 14949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:29.304081 14943 server_base.cc:1034] running on GCE node
I20260928 12:58:29.304363 14943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:29.304644 14943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:29.305831 14943 hybrid_clock.cc:648] HybridClock initialized: now 1790600309305807 us; error 52 us; skew 500 ppm
Sep 28 12:58:29 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600309, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:29.308776 14943 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:29.309392 14943 webserver.cc:460] Webserver started at http://127.11.11.65:36969/ using document root <none> and password file <none>
I20260928 12:58:29.309655 14943 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:29.309731 14943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:29.309901 14943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:29.311111 14943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "8a5aae8a6df94c9ea8b7aa6becc37c4e"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "e345cb7f588cbde8e1bcb1f88f793177"
server_key_iv: "d612fbe18b9e1b4731b163a6723f1a1a"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.311535 14943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "8a5aae8a6df94c9ea8b7aa6becc37c4e"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "e345cb7f588cbde8e1bcb1f88f793177"
server_key_iv: "d612fbe18b9e1b4731b163a6723f1a1a"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.313314 14943 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 12:58:29.314194 14958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:29.314368 14943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:29.314489 14943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "8a5aae8a6df94c9ea8b7aa6becc37c4e"
format_stamp: "Formatted at 2026-09-28 12:58:29 on dist-test-slave-8sjz"
server_key: "e345cb7f588cbde8e1bcb1f88f793177"
server_key_iv: "d612fbe18b9e1b4731b163a6723f1a1a"
server_key_version: "encryptionkey@0"
I20260928 12:58:29.314579 14943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:29.325855 14943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:29.327913 14943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:29.328130 14943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:29.328410 14943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:29.328763 14943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:29.328824 14943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:29.328871 14943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:29.328919 14943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:29.335281 14943 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:42425
I20260928 12:58:29.335332 15071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:42425 every 8 connection(s)
I20260928 12:58:29.335732 14943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:29.337172 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 14943
I20260928 12:58:29.337260 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:29.337491 11309 external_mini_cluster.cc:1428] Setting key c96fe15572a697c2cb969bd2a5531b5d
Sep 28 12:58:29 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600309, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 12:58:29.346884 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:29.336542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47647 (local address 127.11.11.126:45699)
0928 12:58:29.336591 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:29.336595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:29.336810 (+ 215us) server_negotiation.cc:407] Connection header received
0928 12:58:29.337951 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:29.337952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:29.337961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:29.338015 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:29.338769 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.339672 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.340950 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.341200 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.341309 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:29.343742 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:29.343744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:29.343744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:29.343752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:29.345275 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:29.345804 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:29.345806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:29.345807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:29.345872 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:29.346148 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:29.346149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:29.346150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:29.346281 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:29.346363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:29.346689 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 12:58:29.346789 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:29.347064 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:29.336506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45699 (local address 127.11.11.65:47647)
0928 12:58:29.336677 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:29.336723 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:29.337837 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:29.338100 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:29.338104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:29.338457 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:29.338655 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:29.338660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.339826 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.339827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:29.340843 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:29.340846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:29.341323 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:29.341324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:29.341487 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:29.342160 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:29.342171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:29.343628 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:29.345391 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:29.345400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:29.345403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:29.345725 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:29.345942 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:29.345943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:29.345943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:29.346067 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:29.346352 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:29.346360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:29.346516 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 12:58:29.346614 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:58:29.346810 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 12:58:29.347625 15072 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45699
I20260928 12:58:29.347745 15072 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:29.347977 15072 heartbeater.cc:511] Master 127.11.11.126:45699 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
I20260928 12:58:29.348673 14881 ts_manager.cc:194] Registered new tserver with Master: 8a5aae8a6df94c9ea8b7aa6becc37c4e (127.11.11.65:42425)
I20260928 12:58:29.349547 14881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:47647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
W20260928 12:58:29.899475 11309 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.11.66: real 0.561s user 0.000s sys 0.003s
I20260928 12:58:29.899842 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:30.015834 15077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:30.016121 15077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:30.016152 15077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:30.017498 15077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:30.017544 15077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:30.017572 15077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:30.019264 15077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:30.019533 15077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:30.019785 15077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:30.025107 15082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:30.025255 15085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:30.025107 15083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:30.025539 15077 server_base.cc:1034] running on GCE node
I20260928 12:58:30.025812 15077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:30.026074 15077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:30.027228 15077 hybrid_clock.cc:648] HybridClock initialized: now 1790600310027213 us; error 30 us; skew 500 ppm
Sep 28 12:58:30 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600310, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:30.030030 15077 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:30.030494 15077 webserver.cc:460] Webserver started at http://127.11.11.66:44349/ using document root <none> and password file <none>
I20260928 12:58:30.030746 15077 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:30.030818 15077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:30.030967 15077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:30.032148 15077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "f64b4093cdcf4043a9e8c76f0fcc019c"
format_stamp: "Formatted at 2026-09-28 12:58:30 on dist-test-slave-8sjz"
server_key: "14bed86a124d25aab3f333a55eef8dbd"
server_key_iv: "d52a0e2e371094e9d84fdddf6273b01e"
server_key_version: "encryptionkey@0"
I20260928 12:58:30.032542 15077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "f64b4093cdcf4043a9e8c76f0fcc019c"
format_stamp: "Formatted at 2026-09-28 12:58:30 on dist-test-slave-8sjz"
server_key: "14bed86a124d25aab3f333a55eef8dbd"
server_key_iv: "d52a0e2e371094e9d84fdddf6273b01e"
server_key_version: "encryptionkey@0"
I20260928 12:58:30.033943 15077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:30.034744 15092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:30.034919 15077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:30.035027 15077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "f64b4093cdcf4043a9e8c76f0fcc019c"
format_stamp: "Formatted at 2026-09-28 12:58:30 on dist-test-slave-8sjz"
server_key: "14bed86a124d25aab3f333a55eef8dbd"
server_key_iv: "d52a0e2e371094e9d84fdddf6273b01e"
server_key_version: "encryptionkey@0"
I20260928 12:58:30.035100 15077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:30.062047 15077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:30.064757 15077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:30.064942 15077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:30.065217 15077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:30.065557 15077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:30.065610 15077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:30.065649 15077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:30.065676 15077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:30.071655 15077 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:36617
I20260928 12:58:30.071781 15205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:36617 every 8 connection(s)
I20260928 12:58:30.072163 15077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:30.075970 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15077
I20260928 12:58:30.076122 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:30.076336 11309 external_mini_cluster.cc:1428] Setting key 3e94f24038670f8099d9198f74c5a797
Sep 28 12:58:30 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600310, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:30.083333 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:30.073036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:45903 (local address 127.11.11.126:45699)
0928 12:58:30.073846 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:30.073848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:30.073859 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:30.074320 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:30.074321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:30.074333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:30.074399 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:30.075195 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:30.075720 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:30.077224 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:30.077436 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:30.077496 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:30.080135 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:30.080137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:30.080137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:30.080144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:30.081700 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:30.082247 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:30.082248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:30.082248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:30.082307 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:30.082619 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:30.082620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:30.082620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:30.082752 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:30.082803 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:30.083172 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 12:58:30.083255 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":102,"threads_started":1}
I20260928 12:58:30.083623 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:30.072984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45699 (local address 127.11.11.66:45903)
0928 12:58:30.073185 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:30.073250 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:30.074221 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:30.074514 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:30.074519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:30.074891 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:30.075101 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:30.075106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:30.075835 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:30.075837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:30.077138 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:30.077142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:30.077529 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:30.077530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:30.077697 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:30.078269 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:30.078281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:30.080007 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:30.081796 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:30.081805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:30.081808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:30.082169 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:30.082375 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:30.082377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:30.082377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:30.082544 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:30.082821 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:30.082828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:30.083022 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 12:58:30.083154 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 12:58:30.083353 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260928 12:58:30.084241 15206 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45699
I20260928 12:58:30.084362 15206 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:30.084579 15206 heartbeater.cc:511] Master 127.11.11.126:45699 requested a full tablet report, sending...
I20260928 12:58:30.085043 14881 ts_manager.cc:194] Registered new tserver with Master: f64b4093cdcf4043a9e8c76f0fcc019c (127.11.11.66:36617)
I20260928 12:58:30.085440 14881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:45903
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
I20260928 12:58:30.350746 15072 heartbeater.cc:503] Master 127.11.11.126:45699 was elected leader, sending a full tablet report...
I20260928 12:58:31.086470 15206 heartbeater.cc:503] Master 127.11.11.126:45699 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
W20260928 12:58:31.213601 11309 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.11.11.67: real 1.136s user 0.000s sys 0.003s
I20260928 12:58:31.214198 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:31.304011 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:31.304296 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:31.304322 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:31.305604 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:31.305675 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:31.305729 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:31.307386 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:31.307682 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:31.307963 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:31.313275 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:31.313270 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:31.313488 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:31.313756 15212 server_base.cc:1034] running on GCE node
I20260928 12:58:31.314044 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:31.314309 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:31.315495 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790600311315463 us; error 45 us; skew 500 ppm
Sep 28 12:58:31 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:31.324679 15212 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:31.325186 15212 webserver.cc:460] Webserver started at http://127.11.11.67:39707/ using document root <none> and password file <none>
I20260928 12:58:31.325433 15212 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:31.325505 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:31.325667 15212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:31.326786 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "4c70cbc5128a47aaabf03933a661baca"
format_stamp: "Formatted at 2026-09-28 12:58:31 on dist-test-slave-8sjz"
server_key: "9eaf25ecc03c7617522ed86b26fda547"
server_key_iv: "fdf8b9f06825566cd1530893f1c895e3"
server_key_version: "encryptionkey@0"
I20260928 12:58:31.327176 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "4c70cbc5128a47aaabf03933a661baca"
format_stamp: "Formatted at 2026-09-28 12:58:31 on dist-test-slave-8sjz"
server_key: "9eaf25ecc03c7617522ed86b26fda547"
server_key_iv: "fdf8b9f06825566cd1530893f1c895e3"
server_key_version: "encryptionkey@0"
I20260928 12:58:31.328776 15212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:31.329802 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:31.330070 15212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:31.330195 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "4c70cbc5128a47aaabf03933a661baca"
format_stamp: "Formatted at 2026-09-28 12:58:31 on dist-test-slave-8sjz"
server_key: "9eaf25ecc03c7617522ed86b26fda547"
server_key_iv: "fdf8b9f06825566cd1530893f1c895e3"
server_key_version: "encryptionkey@0"
I20260928 12:58:31.330283 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:31.350468 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:31.353387 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:31.353585 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:31.353875 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:31.354256 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:31.354306 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:31.354350 15212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:31.354369 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:31.360313 15212 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:42225
I20260928 12:58:31.360355 15340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:42225 every 8 connection(s)
I20260928 12:58:31.360730 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:31 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:31.369975 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15212
I20260928 12:58:31.370076 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:31.370319 11309 external_mini_cluster.cc:1428] Setting key b4850fc6ea165c3d7804f2410cd78f6d
I20260928 12:58:31.371771 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:31.361663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:60513 (local address 127.11.11.126:45699)
0928 12:58:31.361843 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:31.361845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:31.361942 (+ 97us) server_negotiation.cc:407] Connection header received
0928 12:58:31.362896 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:31.362897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:31.362911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:31.362979 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:31.363659 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.364273 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.365292 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.365493 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.365551 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:31.368016 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:31.368018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:31.368019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:31.368027 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:31.370060 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.370695 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.370698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.370698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.370772 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.371048 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.371048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.371049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.371186 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:31.371270 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:31.371557 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 12:58:31.371673 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260928 12:58:31.371964 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:31.361614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45699 (local address 127.11.11.67:60513)
0928 12:58:31.361803 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:31.361849 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:31.362793 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:31.363057 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:31.363061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:31.363340 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:31.363570 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:31.363574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.364367 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.364369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:31.365202 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:31.365205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.365572 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.365573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:31.365679 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:31.366250 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:31.366259 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:31.367900 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:31.370181 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:31.370191 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:31.370194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:31.370541 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:31.370845 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:31.370846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:31.370846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:31.370972 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:31.371259 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:31.371266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:31.371416 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 12:58:31.371535 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:58:31.371709 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 12:58:31.372485 15341 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45699
I20260928 12:58:31.372573 15341 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:31.372784 15341 heartbeater.cc:511] Master 127.11.11.126:45699 requested a full tablet report, sending...
I20260928 12:58:31.373126 14881 ts_manager.cc:194] Registered new tserver with Master: 4c70cbc5128a47aaabf03933a661baca (127.11.11.67:42225)
I20260928 12:58:31.373538 14881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:60513
I20260928 12:58:31.382179 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:58:31 dist-test-slave-8sjz krb5kdc[14836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:58:31 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:31.417685 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:31.408695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.11.11.65:42425)
0928 12:58:31.408881 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:31.408887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:31.408902 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:58:31.408934 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:31.408937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:31.408956 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:31.409054 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:31.409683 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.410905 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.411723 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.412143 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.412219 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:31.414919 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:31.414928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:31.414928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:31.414936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:31.416552 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.416885 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.416888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.416889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.416945 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.417110 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.417111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.417111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.417243 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:31.417359 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:31.417467 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 12:58:31.417541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
Sep 28 12:58:31 dist-test-slave-8sjz krb5kdc[14836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:31.426239 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:31.419117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.11.11.126:45699)
0928 12:58:31.419166 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:31.419168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:31.419238 (+ 70us) server_negotiation.cc:407] Connection header received
0928 12:58:31.419297 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:31.419298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:31.419307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:31.419351 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:31.419833 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.420412 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.421155 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:31.421365 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:31.421422 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:31.423754 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:31.423756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:31.423756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:31.423766 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:31.425126 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.425495 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.425496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.425496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.425550 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:31.425727 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:31.425728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:31.425728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:31.425915 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:31.426001 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:31.426085 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 12:58:31.426168 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:58:31.426527 14881 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44288
I20260928 12:58:31.426954 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14943
I20260928 12:58:31.431892 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15077
I20260928 12:58:31.436451 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15212
I20260928 12:58:31.440908 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 14849
2026-09-28T12:58:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3667 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:32 dist-test-slave-8sjz krb5kdc[15354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:32 dist-test-slave-8sjz krb5kdc[15354](info): set up 2 sockets
Sep 28 12:58:32 dist-test-slave-8sjz krb5kdc[15354](info): commencing operation
krb5kdc: starting...
W20260928 12:58:33.941193 11309 mini_kdc.cc:121] Time spent starting KDC: real 2.479s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:34.067158 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34141
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:34.149681 15367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:34.149914 15367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:34.149941 15367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:34.151201 15367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:34.151257 15367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:34.151271 15367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:34.151286 15367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:34.151298 15367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:34.152916 15367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:34.153200 15367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:34.153460 15367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:34.158461 15375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.158583 15372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.158468 15373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:34.158815 15367 server_base.cc:1034] running on GCE node
I20260928 12:58:34.159075 15367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:34.159396 15367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:34.160563 15367 hybrid_clock.cc:648] HybridClock initialized: now 1790600314160549 us; error 46 us; skew 500 ppm
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:34.163245 15367 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:34.163605 15367 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:34.163681 15367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:34.163867 15367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:34.165032 15367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "b1787ae0d9114ab6a51076710b5ffdce"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "ebc70734d67be560eb1d8409695d9aec"
server_key_iv: "9992ddbd2111ff3b83461900cfba6bcf"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.165409 15367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "b1787ae0d9114ab6a51076710b5ffdce"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "ebc70734d67be560eb1d8409695d9aec"
server_key_iv: "9992ddbd2111ff3b83461900cfba6bcf"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.166759 15367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:34.167493 15381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.167662 15367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:34.167794 15367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "b1787ae0d9114ab6a51076710b5ffdce"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "ebc70734d67be560eb1d8409695d9aec"
server_key_iv: "9992ddbd2111ff3b83461900cfba6bcf"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.167879 15367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:34.203322 15367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:34.205554 15367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:34.205742 15367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:34.209932 15367 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:34141
I20260928 12:58:34.209975 15433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:34141 every 8 connection(s)
I20260928 12:58:34.210304 15367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:34.211298 15434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:34.212400 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15367
I20260928 12:58:34.212484 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:34.212677 11309 external_mini_cluster.cc:1428] Setting key c1ed2d1efc51cf4ac137ae234377b0c6
I20260928 12:58:34.214419 15434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: Bootstrap starting.
I20260928 12:58:34.215123 15434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:34.215457 15434 log.cc:824] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:34.216279 15434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: No bootstrap required, opened a new log
I20260928 12:58:34.218436 15434 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } }
I20260928 12:58:34.218577 15434 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:34.218600 15434 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1787ae0d9114ab6a51076710b5ffdce, State: Initialized, Role: FOLLOWER
I20260928 12:58:34.218760 15434 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } }
I20260928 12:58:34.218840 15434 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:34.218866 15434 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:34.218940 15434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:34.219496 15434 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } }
I20260928 12:58:34.219624 15434 leader_election.cc:302] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1787ae0d9114ab6a51076710b5ffdce; no voters:
I20260928 12:58:34.219836 15434 leader_election.cc:288] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:34.219866 15439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:34.219997 15439 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [term 1 LEADER]: Becoming Leader. State: Replica: b1787ae0d9114ab6a51076710b5ffdce, State: Running, Role: LEADER
I20260928 12:58:34.220150 15434 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:34.220177 15439 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } }
I20260928 12:58:34.220526 15439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1787ae0d9114ab6a51076710b5ffdce. Latest consensus state: current_term: 1 leader_uuid: "b1787ae0d9114ab6a51076710b5ffdce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } } }
I20260928 12:58:34.220541 15441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1787ae0d9114ab6a51076710b5ffdce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1787ae0d9114ab6a51076710b5ffdce" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34141 } } }
I20260928 12:58:34.220628 15439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:34.220649 15441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:34.221030 15446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:34.221577 15446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:34.224432 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.213261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.11.11.126:34141)
0928 12:58:34.213460 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.213470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.213505 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:58:34.214108 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.214118 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.214186 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.214358 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:34.214981 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.216440 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.217213 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.217898 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.218035 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.220868 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:34.220873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:34.220873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:34.220886 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:34.222789 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.223222 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.223225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.223225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.223307 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.223523 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.223523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.223523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.223761 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:34.223870 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.224022 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.224233 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 12:58:34.225345 15446 catalog_manager.cc:1347] Generated new cluster ID: 467f5f19de8c48a48581ed68fa16c555
I20260928 12:58:34.225419 15446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:34.242365 15446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:34.242990 15446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:34.245534 15446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: Generated new TSK 0
I20260928 12:58:34.245744 15446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:34.279663 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:34.386878 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:34.387113 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:34.387141 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:34.388401 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:34.388446 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:34.388475 15460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:34.390069 15460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:34.390347 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:34.390606 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:34.395609 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.395622 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.395771 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:34.396174 15460 server_base.cc:1034] running on GCE node
I20260928 12:58:34.396418 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:34.396682 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:34.397866 15460 hybrid_clock.cc:648] HybridClock initialized: now 1790600314397855 us; error 40 us; skew 500 ppm
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:34.400554 15460 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:34.400951 15460 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:34.401028 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:34.401176 15460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:34.402349 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "edca9a19e24e4768a0148f60a66b7776"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "8e7449e109f5590e57a63dd98fd00b9f"
server_key_iv: "8979e3799bb8bc0530ef2133551a2937"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.402734 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "edca9a19e24e4768a0148f60a66b7776"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "8e7449e109f5590e57a63dd98fd00b9f"
server_key_iv: "8979e3799bb8bc0530ef2133551a2937"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.404225 15460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:58:34.405007 15474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.405205 15460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:34.405313 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "edca9a19e24e4768a0148f60a66b7776"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "8e7449e109f5590e57a63dd98fd00b9f"
server_key_iv: "8979e3799bb8bc0530ef2133551a2937"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.405402 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:34.431032 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:34.433640 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:34.433826 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:34.434089 15460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:34.434433 15460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:34.434490 15460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.434536 15460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:34.434561 15460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.440500 15460 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:45723
I20260928 12:58:34.440582 15587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:45723 every 8 connection(s)
I20260928 12:58:34.440950 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:34.446331 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15460
I20260928 12:58:34.446456 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:34.446693 11309 external_mini_cluster.cc:1428] Setting key a45e63cb23df73247d8c17f3a5fa21b5
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:34.452276 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.441808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45599 (local address 127.11.11.126:34141)
0928 12:58:34.441880 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.441882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.442112 (+ 230us) server_negotiation.cc:407] Connection header received
0928 12:58:34.442670 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.442672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.442685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.442725 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:34.443767 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.444386 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.445438 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.445662 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.445757 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.448827 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:34.448829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:34.448829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:34.448836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:34.450569 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.451028 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.451030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.451030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.451100 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.451477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.451478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.451478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.451640 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:34.451720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.451996 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.452132 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:58:34.452347 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.441793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34141 (local address 127.11.11.65:45599)
0928 12:58:34.441973 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.442019 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.442555 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.442808 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.442812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.443105 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:34.443644 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.443650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.444487 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.444489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.445335 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.445337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.445769 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.445770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.445904 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.446607 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:34.446616 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:34.448717 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:34.450705 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.450713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.450715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.450944 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.451196 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.451197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.451197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.451395 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.451719 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:34.451724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:34.451854 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.451942 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.452141 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260928 12:58:34.453069 15588 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34141
I20260928 12:58:34.453195 15588 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:34.453372 15588 heartbeater.cc:511] Master 127.11.11.126:34141 requested a full tablet report, sending...
I20260928 12:58:34.453940 15398 ts_manager.cc:194] Registered new tserver with Master: edca9a19e24e4768a0148f60a66b7776 (127.11.11.65:45723)
I20260928 12:58:34.454561 15398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:45599
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:34.475297 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:34.558921 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:34.559144 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:34.559170 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:34.560484 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:34.560530 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:34.560581 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:34.562288 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:34.562558 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:34.562808 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:34.568122 15599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.567994 15601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.567994 15598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:34.568413 15593 server_base.cc:1034] running on GCE node
I20260928 12:58:34.568624 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:34.568928 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:34.570099 15593 hybrid_clock.cc:648] HybridClock initialized: now 1790600314570093 us; error 37 us; skew 500 ppm
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:34.572666 15593 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:34.573037 15593 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:34.573109 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:34.573266 15593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:34.574357 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "30167e2f40bc4d97bad29d23bf70f554"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "4b8501cdc03b57e60fd05e876b336903"
server_key_iv: "f02917f109984f8b25f0cddd867b253f"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.574751 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "30167e2f40bc4d97bad29d23bf70f554"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "4b8501cdc03b57e60fd05e876b336903"
server_key_iv: "f02917f109984f8b25f0cddd867b253f"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.576193 15593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:34.576947 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.577123 15593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:34.577229 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "30167e2f40bc4d97bad29d23bf70f554"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "4b8501cdc03b57e60fd05e876b336903"
server_key_iv: "f02917f109984f8b25f0cddd867b253f"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.577311 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:34.594769 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:34.596818 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:34.596988 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:34.597234 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:34.597559 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:34.597615 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.597661 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:34.597687 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.603705 15593 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:43251
I20260928 12:58:34.603793 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:43251 every 8 connection(s)
I20260928 12:58:34.604136 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:34.610706 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15593
I20260928 12:58:34.610829 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:34.611011 11309 external_mini_cluster.cc:1428] Setting key 61af2be7ea117dcc25fa74ad41194329
I20260928 12:58:34.614437 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.604950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:36173 (local address 127.11.11.126:34141)
0928 12:58:34.605048 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.605051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.605265 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:58:34.605735 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.605736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.605748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.605810 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:34.606890 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.607385 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.608492 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.608680 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.608740 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.611161 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:34.611162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:34.611163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:34.611176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:34.612958 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.613433 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.613435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.613435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.613502 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.613782 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.613783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.613783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.613905 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:34.613985 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.614300 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.614369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 12:58:34.614758 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.604932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34141 (local address 127.11.11.66:36173)
0928 12:58:34.605099 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.605144 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.605642 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.605866 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.605870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.606141 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:34.606796 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.606803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.607466 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.607467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.608416 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.608419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.608796 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.608797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.608914 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.609547 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:34.609556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:34.611062 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:34.613047 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.613053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.613056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.613347 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.613572 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.613573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.613574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.613701 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.613973 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:34.613981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:34.614136 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.614270 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.614463 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 12:58:34.615263 15721 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34141
I20260928 12:58:34.615375 15721 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:34.615550 15721 heartbeater.cc:511] Master 127.11.11.126:34141 requested a full tablet report, sending...
I20260928 12:58:34.615967 15398 ts_manager.cc:194] Registered new tserver with Master: 30167e2f40bc4d97bad29d23bf70f554 (127.11.11.66:43251)
I20260928 12:58:34.616389 15398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:36173
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:34.639707 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:34.723376 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:34.723623 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:34.723649 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:34.725175 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:34.725232 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:34.725263 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:34.726879 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:34.727159 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:34.727411 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:34.732496 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.732489 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:34.732489 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:34.732864 15726 server_base.cc:1034] running on GCE node
I20260928 12:58:34.733062 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:34.733274 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:34.734413 15726 hybrid_clock.cc:648] HybridClock initialized: now 1790600314734396 us; error 35 us; skew 500 ppm
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:34.737074 15726 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:34.737452 15726 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:34.737502 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:34.737602 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:34.738639 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "c7e5b89511834325a2fd2a8c68cf9e1a"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "dfc9d3864206936209b02bc27de00a45"
server_key_iv: "d0f378bb0eaf5e018d3e47279d135702"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.738963 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "c7e5b89511834325a2fd2a8c68cf9e1a"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "dfc9d3864206936209b02bc27de00a45"
server_key_iv: "d0f378bb0eaf5e018d3e47279d135702"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.740340 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:34.741113 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.741292 15726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:34.741361 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "c7e5b89511834325a2fd2a8c68cf9e1a"
format_stamp: "Formatted at 2026-09-28 12:58:34 on dist-test-slave-8sjz"
server_key: "dfc9d3864206936209b02bc27de00a45"
server_key_iv: "d0f378bb0eaf5e018d3e47279d135702"
server_key_version: "encryptionkey@0"
I20260928 12:58:34.741416 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:34.760347 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:34.762806 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:34.762928 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:34.763127 15726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:34.763466 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:34.763505 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.763527 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:34.763553 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:34.769706 15726 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:33421
I20260928 12:58:34.769735 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:33421 every 8 connection(s)
I20260928 12:58:34.770027 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:34.775170 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15726
I20260928 12:58:34.775265 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:34.775475 11309 external_mini_cluster.cc:1428] Setting key f5e3f9ac682cb948239a01e857ca206f
Sep 28 12:58:34 dist-test-slave-8sjz krb5kdc[15354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600314, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:34.779753 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.770765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:43433 (local address 127.11.11.126:34141)
0928 12:58:34.770813 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.770815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.771068 (+ 253us) server_negotiation.cc:407] Connection header received
0928 12:58:34.771543 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.771544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.771554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.771608 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:34.772622 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.773112 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.774256 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.774614 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.774750 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.777177 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:34.777178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:34.777179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:34.777193 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:34.778452 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.778858 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.778858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.778859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.778914 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.779135 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.779136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.779136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.779245 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:34.779295 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.779588 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.779674 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:34.779883 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.770791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34141 (local address 127.11.11.67:43433)
0928 12:58:34.770944 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.770981 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.771448 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.771665 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.771668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.771934 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:34.772520 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.772526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.773302 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.773303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.774166 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.774168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.774782 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.774782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.774892 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.775582 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:34.775595 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:34.777072 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:34.778533 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.778539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.778541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.778789 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.778974 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:34.778975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:34.778975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:34.779082 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:34.779299 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:34.779303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:34.779429 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.779517 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.779652 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260928 12:58:34.780396 15854 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34141
I20260928 12:58:34.780521 15854 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:34.780715 15854 heartbeater.cc:511] Master 127.11.11.126:34141 requested a full tablet report, sending...
I20260928 12:58:34.781131 15398 ts_manager.cc:194] Registered new tserver with Master: c7e5b89511834325a2fd2a8c68cf9e1a (127.11.11.67:33421)
I20260928 12:58:34.781558 15398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:43433
I20260928 12:58:34.787295 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:58:34.794369 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.788677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.11.11.126:34141)
0928 12:58:34.788763 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.788766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.788833 (+ 67us) server_negotiation.cc:407] Connection header received
0928 12:58:34.788881 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.788882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.788892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.788965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:34.789509 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.789988 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.790661 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.790838 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.790888 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.792157 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:34.792158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:34.792158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:34.792163 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:34.793405 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.793721 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.793722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.793722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.793773 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:34.793938 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:34.793938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:34.793938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:34.794078 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:34.794152 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.794242 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.794301 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:58:34.796066 15398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:34.798305 15398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:34.805984 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:43251 (local address 127.0.0.1:49606)
0928 12:58:34.801500 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.801509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.801544 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.801997 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.801997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.801999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:34.802042 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.802044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803485 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.803485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.804758 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.804762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.805749 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.805750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.805814 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.805817 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.805868 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.805895 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":122,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260928 12:58:34.806033 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.11.11.66:43251)
0928 12:58:34.801831 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.801836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.801846 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:34.801885 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.801887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.801901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.802113 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:34.802194 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803329 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.804869 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.805632 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.805747 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.805811 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.805900 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.805935 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260928 12:58:34.806185 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.11.11.65:45723)
0928 12:58:34.802127 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.802132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.802249 (+ 117us) server_negotiation.cc:407] Connection header received
0928 12:58:34.802336 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.802338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.802353 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.802453 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:34.802567 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803268 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.804996 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.805527 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.805622 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.805690 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.806026 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.806076 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
I20260928 12:58:34.805984 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:45723 (local address 127.0.0.1:44696)
0928 12:58:34.802182 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.802192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.802258 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.802449 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.802450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.802452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:34.802507 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.802509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803380 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.803381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.804904 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.804905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.805621 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.805623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.805715 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.805732 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.805806 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.805876 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":67,"thread_start_us":77,"threads_started":1}
I20260928 12:58:34.807147 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33421 (local address 127.0.0.1:34392)
0928 12:58:34.801248 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:34.801285 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:34.801347 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:34.801878 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:34.801882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:34.801885 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:34.801982 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.801985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803656 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.803657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.804870 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:34.804873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.806627 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.806895 (+ 268us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:34.806970 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.806975 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:34.807026 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:58:34.807074 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":6,"thread_start_us":82,"threads_started":1}
I20260928 12:58:34.807189 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:34.801513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.11.11.67:33421)
0928 12:58:34.801672 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:34.801676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:34.801685 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:34.801716 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:34.801718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:34.801732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:34.802067 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:34.802154 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.803329 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.804958 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:34.805644 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:34.805747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:34.805815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:34.807061 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0928 12:58:34.807110 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 12:58:34.807350 15522 tablet_service.cc:1467] Processing CreateTablet for tablet db7c59b9bcde4fc79820842c918bedbc (DEFAULT_TABLE table=test-table [id=614148a1211e4373a1a47228426eb468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:34.807350 15655 tablet_service.cc:1467] Processing CreateTablet for tablet db7c59b9bcde4fc79820842c918bedbc (DEFAULT_TABLE table=test-table [id=614148a1211e4373a1a47228426eb468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:34.807744 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c59b9bcde4fc79820842c918bedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:34.807744 15522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c59b9bcde4fc79820842c918bedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:34.808208 15788 tablet_service.cc:1467] Processing CreateTablet for tablet db7c59b9bcde4fc79820842c918bedbc (DEFAULT_TABLE table=test-table [id=614148a1211e4373a1a47228426eb468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:34.808532 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c59b9bcde4fc79820842c918bedbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:34.811048 15871 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Bootstrap starting.
I20260928 12:58:34.811611 15872 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Bootstrap starting.
I20260928 12:58:34.811970 15871 tablet_bootstrap.cc:654] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:34.812265 15872 tablet_bootstrap.cc:654] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:34.812561 15871 log.cc:824] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:34.812822 15872 log.cc:824] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:34.813201 15873 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Bootstrap starting.
I20260928 12:58:34.813341 15871 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: No bootstrap required, opened a new log
I20260928 12:58:34.813417 15871 ts_tablet_manager.cc:1397] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:34.813637 15872 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: No bootstrap required, opened a new log
I20260928 12:58:34.813707 15872 ts_tablet_manager.cc:1397] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:34.813884 15873 tablet_bootstrap.cc:654] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:34.814553 15873 log.cc:824] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:34.815318 15871 raft_consensus.cc:348] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.815445 15873 tablet_bootstrap.cc:492] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: No bootstrap required, opened a new log
I20260928 12:58:34.815505 15871 raft_consensus.cc:374] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:34.815528 15873 ts_tablet_manager.cc:1397] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:34.815547 15871 raft_consensus.cc:729] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e5b89511834325a2fd2a8c68cf9e1a, State: Initialized, Role: FOLLOWER
I20260928 12:58:34.815687 15871 consensus_queue.cc:260] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.815994 15871 ts_tablet_manager.cc:1428] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:34.816148 15854 heartbeater.cc:503] Master 127.11.11.126:34141 was elected leader, sending a full tablet report...
I20260928 12:58:34.816061 15872 raft_consensus.cc:348] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.816220 15872 raft_consensus.cc:374] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:34.816253 15872 raft_consensus.cc:729] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30167e2f40bc4d97bad29d23bf70f554, State: Initialized, Role: FOLLOWER
I20260928 12:58:34.816372 15872 consensus_queue.cc:260] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.816638 15872 ts_tablet_manager.cc:1428] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:34.816764 15721 heartbeater.cc:503] Master 127.11.11.126:34141 was elected leader, sending a full tablet report...
I20260928 12:58:34.817924 15873 raft_consensus.cc:348] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.818075 15873 raft_consensus.cc:374] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:34.818097 15873 raft_consensus.cc:729] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edca9a19e24e4768a0148f60a66b7776, State: Initialized, Role: FOLLOWER
I20260928 12:58:34.818235 15873 consensus_queue.cc:260] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:34.818512 15873 ts_tablet_manager.cc:1428] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:34.818547 15588 heartbeater.cc:503] Master 127.11.11.126:34141 was elected leader, sending a full tablet report...
W20260928 12:58:34.854703 15722 tablet.cc:2369] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:58:34.941759 15589 tablet.cc:2369] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:35.020025 15879 raft_consensus.cc:482] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:35.020226 15879 raft_consensus.cc:504] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
W20260928 12:58:35.020550 15855 tablet.cc:2369] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:35.020689 15879 leader_election.cc:288] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e5b89511834325a2fd2a8c68cf9e1a (127.11.11.67:33421), 30167e2f40bc4d97bad29d23bf70f554 (127.11.11.66:43251)
I20260928 12:58:35.023831 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.020964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45689 (local address 127.11.11.66:43251)
0928 12:58:35.021012 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.021015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.021148 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:58:35.021200 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.021201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.021213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.021268 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:35.021444 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.021958 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023012 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.023437 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023495 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.023502 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.023693 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.023752 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:35.023860 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.020882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:43251 (local address 127.11.11.65:45689)
0928 12:58:35.021062 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.021074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.021142 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.021279 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.021280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.021282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:35.021365 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:35.021367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.022085 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.022086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:35.022916 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:35.022918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.023515 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:35.023615 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.023621 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.023677 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.023740 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260928 12:58:35.023895 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.021032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59573 (local address 127.11.11.67:33421)
0928 12:58:35.021081 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.021083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.021224 (+ 141us) server_negotiation.cc:407] Connection header received
0928 12:58:35.021280 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.021281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.021291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.021345 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:35.021535 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.022158 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023121 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.023513 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023600 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.023609 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.023786 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.023825 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:58:35.024102 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.020882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33421 (local address 127.11.11.65:59573)
0928 12:58:35.021075 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.021087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.021195 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.021364 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.021364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.021366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:35.021461 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:35.021463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.022257 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.022258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:35.023034 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:35.023035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.023607 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.023608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:35.023705 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.023709 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.023768 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.023808 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260928 12:58:35.024277 15675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c59b9bcde4fc79820842c918bedbc" candidate_uuid: "edca9a19e24e4768a0148f60a66b7776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30167e2f40bc4d97bad29d23bf70f554" is_pre_election: true
I20260928 12:58:35.024485 15675 raft_consensus.cc:2454] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edca9a19e24e4768a0148f60a66b7776 in term 0.
I20260928 12:58:35.024518 15808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c59b9bcde4fc79820842c918bedbc" candidate_uuid: "edca9a19e24e4768a0148f60a66b7776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" is_pre_election: true
I20260928 12:58:35.024691 15808 raft_consensus.cc:2454] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edca9a19e24e4768a0148f60a66b7776 in term 0.
I20260928 12:58:35.024760 15475 leader_election.cc:302] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30167e2f40bc4d97bad29d23bf70f554, edca9a19e24e4768a0148f60a66b7776; no voters:
I20260928 12:58:35.024884 15879 raft_consensus.cc:2781] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:35.024935 15879 raft_consensus.cc:482] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:35.024962 15879 raft_consensus.cc:3037] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.025704 15879 raft_consensus.cc:504] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:35.025955 15879 leader_election.cc:288] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [CANDIDATE]: Term 1 election: Requested vote from peers c7e5b89511834325a2fd2a8c68cf9e1a (127.11.11.67:33421), 30167e2f40bc4d97bad29d23bf70f554 (127.11.11.66:43251)
I20260928 12:58:35.026054 15808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c59b9bcde4fc79820842c918bedbc" candidate_uuid: "edca9a19e24e4768a0148f60a66b7776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a"
I20260928 12:58:35.026155 15808 raft_consensus.cc:3037] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.026875 15808 raft_consensus.cc:2454] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edca9a19e24e4768a0148f60a66b7776 in term 1.
I20260928 12:58:35.027062 15478 leader_election.cc:302] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e5b89511834325a2fd2a8c68cf9e1a, edca9a19e24e4768a0148f60a66b7776; no voters:
I20260928 12:58:35.027096 15675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c59b9bcde4fc79820842c918bedbc" candidate_uuid: "edca9a19e24e4768a0148f60a66b7776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30167e2f40bc4d97bad29d23bf70f554"
I20260928 12:58:35.027154 15879 raft_consensus.cc:2781] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:35.027182 15675 raft_consensus.cc:3037] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.027320 15879 raft_consensus.cc:686] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [term 1 LEADER]: Becoming Leader. State: Replica: edca9a19e24e4768a0148f60a66b7776, State: Running, Role: LEADER
I20260928 12:58:35.027427 15879 consensus_queue.cc:237] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } }
I20260928 12:58:35.028407 15675 raft_consensus.cc:2454] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edca9a19e24e4768a0148f60a66b7776 in term 1.
I20260928 12:58:35.029063 15397 catalog_manager.cc:5495] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 reported cstate change: term changed from 0 to 1, leader changed from <none> to edca9a19e24e4768a0148f60a66b7776 (127.11.11.65). New cstate: current_term: 1 leader_uuid: "edca9a19e24e4768a0148f60a66b7776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edca9a19e24e4768a0148f60a66b7776" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 45723 } health_report { overall_health: HEALTHY } } }
I20260928 12:58:35.035621 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.032376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.11.11.65:45723)
0928 12:58:35.032425 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.032428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.032437 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:35.032475 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.032476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.032498 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.032578 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:35.032716 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.033460 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.034759 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.035101 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.035160 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.035316 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:35.035420 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:35.035478 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.035490 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.035507 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:35.037607 15675 raft_consensus.cc:1261] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 1 FOLLOWER]: Refusing update from remote peer edca9a19e24e4768a0148f60a66b7776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:35.037607 15808 raft_consensus.cc:1261] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 1 FOLLOWER]: Refusing update from remote peer edca9a19e24e4768a0148f60a66b7776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:35.037993 15879 consensus_queue.cc:1035] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e5b89511834325a2fd2a8c68cf9e1a" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:35.038143 15882 consensus_queue.cc:1035] T db7c59b9bcde4fc79820842c918bedbc P edca9a19e24e4768a0148f60a66b7776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30167e2f40bc4d97bad29d23bf70f554" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:35.040364 15885 mvcc.cc:205] Tried to move back new op lower bound from 7334298890391134208 to 7334298890353033216. Current Snapshot: MvccSnapshot[applied={T|T < 7334298890391134208}]
I20260928 12:58:35.041013 15887 mvcc.cc:205] Tried to move back new op lower bound from 7334298890391134208 to 7334298890353033216. Current Snapshot: MvccSnapshot[applied={T|T < 7334298890391134208 or (T in {7334298890391134208})}]
I20260928 12:58:35.041349 15886 mvcc.cc:205] Tried to move back new op lower bound from 7334298890391134208 to 7334298890353033216. Current Snapshot: MvccSnapshot[applied={T|T < 7334298890391134208}]
I20260928 12:58:35.043511 15397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:35.043610 15397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:35.044478 15397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1787ae0d9114ab6a51076710b5ffdce: Sending DeleteTablet for 3 replicas of tablet db7c59b9bcde4fc79820842c918bedbc
I20260928 12:58:35.044865 15522 tablet_service.cc:1514] Processing DeleteTablet for tablet db7c59b9bcde4fc79820842c918bedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:44696
I20260928 12:58:35.044863 15655 tablet_service.cc:1514] Processing DeleteTablet for tablet db7c59b9bcde4fc79820842c918bedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:49606
I20260928 12:58:35.045122 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15460
I20260928 12:58:35.045147 15788 tablet_service.cc:1514] Processing DeleteTablet for tablet db7c59b9bcde4fc79820842c918bedbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34392
I20260928 12:58:35.047029 15897 tablet_replica.cc:330] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: stopping tablet replica
I20260928 12:58:35.047164 15897 raft_consensus.cc:2229] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:35.047312 15897 raft_consensus.cc:2258] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:35.047499 15896 tablet_replica.cc:330] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: stopping tablet replica
I20260928 12:58:35.047607 15896 raft_consensus.cc:2229] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:35.047715 15896 raft_consensus.cc:2258] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:35.047780 15897 ts_tablet_manager.cc:1905] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:35.048187 15896 ts_tablet_manager.cc:1905] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:35.049543 15897 ts_tablet_manager.cc:1918] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:35.049633 15897 log.cc:1196] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/db7c59b9bcde4fc79820842c918bedbc
I20260928 12:58:35.049923 15897 ts_tablet_manager.cc:1939] T db7c59b9bcde4fc79820842c918bedbc P c7e5b89511834325a2fd2a8c68cf9e1a: Deleting consensus metadata
I20260928 12:58:35.050136 15896 ts_tablet_manager.cc:1918] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:35.050216 15896 log.cc:1196] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/db7c59b9bcde4fc79820842c918bedbc
I20260928 12:58:35.050422 15385 catalog_manager.cc:4841] TS c7e5b89511834325a2fd2a8c68cf9e1a (127.11.11.67:33421): tablet db7c59b9bcde4fc79820842c918bedbc (table test-table [id=614148a1211e4373a1a47228426eb468]) successfully deleted
I20260928 12:58:35.050508 15896 ts_tablet_manager.cc:1939] T db7c59b9bcde4fc79820842c918bedbc P 30167e2f40bc4d97bad29d23bf70f554: Deleting consensus metadata
I20260928 12:58:35.050931 15382 catalog_manager.cc:4841] TS 30167e2f40bc4d97bad29d23bf70f554 (127.11.11.66:43251): tablet db7c59b9bcde4fc79820842c918bedbc (table test-table [id=614148a1211e4373a1a47228426eb468]) successfully deleted
W20260928 12:58:35.051658 15385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:45723 (error 108)
I20260928 12:58:35.051956 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15593
I20260928 12:58:35.052862 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.052157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:45723 (local address 127.0.0.1:44718)
0928 12:58:35.052790 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.052832 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:45723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612}
W20260928 12:58:35.053032 15385 catalog_manager.cc:4584] TS edca9a19e24e4768a0148f60a66b7776 (127.11.11.65:45723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db7c59b9bcde4fc79820842c918bedbc: Network error: Client connection negotiation failed: client connection to 127.11.11.65:45723: connect: Connection refused (error 111)
I20260928 12:58:35.057806 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15726
I20260928 12:58:35.062355 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15367
2026-09-28T12:58:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3621 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T12:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:35Z Disabled control of system clock
I20260928 12:58:35.091208 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46401
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:58:35.173870 15902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:35.174106 15902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:35.174129 15902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:35.175328 15902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:35.175383 15902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:35.175408 15902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:35.175424 15902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:35.175436 15902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:35.176800 15902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:35.177009 15902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:35.177273 15902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:35.182402 15908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.182298 15910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.182294 15907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:35.182665 15902 server_base.cc:1034] running on GCE node
I20260928 12:58:35.182973 15902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:35.183296 15902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:35.184469 15902 hybrid_clock.cc:648] HybridClock initialized: now 1790600315184455 us; error 47 us; skew 500 ppm
I20260928 12:58:35.185819 15902 webserver.cc:460] Webserver started at http://127.11.11.126:46803/ using document root <none> and password file <none>
I20260928 12:58:35.186065 15902 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:35.186134 15902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:35.186304 15902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:35.187419 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "f32b88cbf11440fb84174002ec8b57c5"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "396b6b74d82df7a13e2d738eac2814b8"
server_key_iv: "e5e124ba9670f9cfcaffe4e1ea822fc6"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.187791 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "f32b88cbf11440fb84174002ec8b57c5"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "396b6b74d82df7a13e2d738eac2814b8"
server_key_iv: "e5e124ba9670f9cfcaffe4e1ea822fc6"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.189179 15902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:35.189939 15916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.190114 15902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:35.190224 15902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "f32b88cbf11440fb84174002ec8b57c5"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "396b6b74d82df7a13e2d738eac2814b8"
server_key_iv: "e5e124ba9670f9cfcaffe4e1ea822fc6"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.190305 15902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:35.193830 15902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:35.194120 15902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:35.194267 15902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:35.198297 15902 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46401
I20260928 12:58:35.198329 15968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46401 every 8 connection(s)
I20260928 12:58:35.198684 15902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:35.199395 15969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:35.202275 15969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: Bootstrap starting.
I20260928 12:58:35.202826 15969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:35.203064 15969 log.cc:824] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:35.203720 15969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: No bootstrap required, opened a new log
I20260928 12:58:35.205387 15969 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } }
I20260928 12:58:35.205538 15969 raft_consensus.cc:374] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:35.205560 15969 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32b88cbf11440fb84174002ec8b57c5, State: Initialized, Role: FOLLOWER
I20260928 12:58:35.205710 15969 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } }
I20260928 12:58:35.205796 15969 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:35.205821 15969 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:35.205891 15969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.205912 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15902
I20260928 12:58:35.206013 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:35.206203 11309 external_mini_cluster.cc:1428] Setting key 1341415ef207dd8b140759a486023e92
I20260928 12:58:35.206743 15969 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } }
I20260928 12:58:35.206866 15969 leader_election.cc:302] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32b88cbf11440fb84174002ec8b57c5; no voters:
I20260928 12:58:35.207051 15969 leader_election.cc:288] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:35.207082 15974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:35.207231 15974 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [term 1 LEADER]: Becoming Leader. State: Replica: f32b88cbf11440fb84174002ec8b57c5, State: Running, Role: LEADER
I20260928 12:58:35.207336 15974 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } }
I20260928 12:58:35.207482 15969 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:35.207726 15974 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32b88cbf11440fb84174002ec8b57c5. Latest consensus state: current_term: 1 leader_uuid: "f32b88cbf11440fb84174002ec8b57c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } } }
I20260928 12:58:35.207806 15974 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:35.208068 15978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:35.208158 15975 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f32b88cbf11440fb84174002ec8b57c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32b88cbf11440fb84174002ec8b57c5" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46401 } } }
I20260928 12:58:35.208297 15975 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:35.208602 15978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:35.209420 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.206768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.11.11.126:46401)
0928 12:58:35.207004 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.207015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.207051 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:58:35.207187 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.207196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.207250 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.208241 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.208682 (+ 441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.208686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.208687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.208691 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.208753 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.208771 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.209101 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.209139 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.209211 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 12:58:35.210683 15978 catalog_manager.cc:1347] Generated new cluster ID: 55d573fd5dd54c22822efd7cd5494356
I20260928 12:58:35.210748 15978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:35.239187 15978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:35.239403 15978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:35.243466 15978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: Generated new TSK 0
I20260928 12:58:35.243702 15978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:35.247720 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--builtin_ntp_servers=127.11.11.84:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:58:35.332280 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:35.332528 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:35.332564 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:35.333791 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:35.333858 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:35.333905 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:35.335294 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:35.335565 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:35.335878 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:35.340942 15998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.340956 15999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.340929 16001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:35.341410 15993 server_base.cc:1034] running on GCE node
I20260928 12:58:35.341637 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:35.341897 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:35.343067 15993 hybrid_clock.cc:648] HybridClock initialized: now 1790600315343053 us; error 28 us; skew 500 ppm
I20260928 12:58:35.344576 15993 webserver.cc:460] Webserver started at http://127.11.11.65:36893/ using document root <none> and password file <none>
I20260928 12:58:35.344802 15993 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:35.344848 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:35.344938 15993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:35.345983 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "1319978e17cc4413bcc1a7f364bbc2ef"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "e70b03c322a2ed9f9fb42910e2a27426"
server_key_iv: "fe75bc95fac63cb842594882a8ea4961"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.346369 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "1319978e17cc4413bcc1a7f364bbc2ef"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "e70b03c322a2ed9f9fb42910e2a27426"
server_key_iv: "fe75bc95fac63cb842594882a8ea4961"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.347669 15993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:35.348546 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.348743 15993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:35.348860 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "1319978e17cc4413bcc1a7f364bbc2ef"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "e70b03c322a2ed9f9fb42910e2a27426"
server_key_iv: "fe75bc95fac63cb842594882a8ea4961"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.348937 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:35.352313 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:35.352557 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:35.352707 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:35.352923 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:35.353243 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:35.353276 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.353328 15993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:35.353351 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.358877 15993 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:41091
I20260928 12:58:35.358961 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:41091 every 8 connection(s)
I20260928 12:58:35.359311 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:35.362474 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.360129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:58325 (local address 127.11.11.126:46401)
0928 12:58:35.360204 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.360207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.360451 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:58:35.360956 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.360957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.360965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.361016 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.362266 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.362267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.362267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.362272 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.362288 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.362292 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.362341 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.362401 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.362417 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:35.362650 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.360123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46401 (local address 127.11.11.65:58325)
0928 12:58:35.360347 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.360383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.360868 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.361077 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.361081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.361088 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.362111 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.362111 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.362141 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.362172 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.362337 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.362340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.362342 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.362395 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.362446 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
I20260928 12:58:35.363013 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 15993
I20260928 12:58:35.363118 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:35.363322 11309 external_mini_cluster.cc:1428] Setting key cd2129e90888c7b5b59e033ac8885e0c
I20260928 12:58:35.363300 16121 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46401
I20260928 12:58:35.363411 16121 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:35.363580 16121 heartbeater.cc:511] Master 127.11.11.126:46401 requested a full tablet report, sending...
I20260928 12:58:35.364200 15932 ts_manager.cc:194] Registered new tserver with Master: 1319978e17cc4413bcc1a7f364bbc2ef (127.11.11.65:41091)
I20260928 12:58:35.364396 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--builtin_ntp_servers=127.11.11.84:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:58:35.448918 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:35.449190 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:35.449215 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:35.450457 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:35.450507 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:35.450567 16124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:35.452080 16124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:35.452358 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:35.452690 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:35.457971 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.457971 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.458019 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:35.458544 16124 server_base.cc:1034] running on GCE node
I20260928 12:58:35.458770 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:35.458982 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:35.460095 16124 hybrid_clock.cc:648] HybridClock initialized: now 1790600315460077 us; error 32 us; skew 500 ppm
I20260928 12:58:35.461452 16124 webserver.cc:460] Webserver started at http://127.11.11.66:35651/ using document root <none> and password file <none>
I20260928 12:58:35.461740 16124 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:35.461787 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:35.461946 16124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:35.463074 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "763a47972e854f91b4d7cbb576c2b467"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "8bc7dcd23fab3a68829d6c932fd7eb7f"
server_key_iv: "8a6e758792e19b2c3eddab7dd9294685"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.463469 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "763a47972e854f91b4d7cbb576c2b467"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "8bc7dcd23fab3a68829d6c932fd7eb7f"
server_key_iv: "8a6e758792e19b2c3eddab7dd9294685"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.464972 16124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:35.465960 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.466197 16124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:35.466300 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "763a47972e854f91b4d7cbb576c2b467"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "8bc7dcd23fab3a68829d6c932fd7eb7f"
server_key_iv: "8a6e758792e19b2c3eddab7dd9294685"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.466379 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:35.469976 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:35.470276 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:35.470427 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:35.470636 16124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:35.470971 16124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:35.471027 16124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.471071 16124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:35.471097 16124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.476980 16124 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:40287
I20260928 12:58:35.477022 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:40287 every 8 connection(s)
I20260928 12:58:35.477396 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:35.479511 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16124
I20260928 12:58:35.479625 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:35.479840 11309 external_mini_cluster.cc:1428] Setting key a1edf6f815811042a8b746b905fdc155
I20260928 12:58:35.480904 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.478200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:40383 (local address 127.11.11.126:46401)
0928 12:58:35.478274 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.478276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.478510 (+ 234us) server_negotiation.cc:407] Connection header received
0928 12:58:35.478993 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.478994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.479000 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.479056 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.480573 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.480575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.480575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.480580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.480600 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.480607 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.480657 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.480814 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.480831 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:58:35.481119 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.478187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46401 (local address 127.11.11.66:40383)
0928 12:58:35.478380 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.478416 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.478899 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.479112 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.479115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.479123 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.480395 (+ 1272us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.480396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.480429 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.480464 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.480697 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.480706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.480709 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.480803 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.480874 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260928 12:58:35.481148 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--builtin_ntp_servers=127.11.11.84:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 12:58:35.481575 16252 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46401
I20260928 12:58:35.481688 16252 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:35.481853 16252 heartbeater.cc:511] Master 127.11.11.126:46401 requested a full tablet report, sending...
I20260928 12:58:35.482204 15932 ts_manager.cc:194] Registered new tserver with Master: 763a47972e854f91b4d7cbb576c2b467 (127.11.11.66:40287)
W20260928 12:58:35.564482 16255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:35.564724 16255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:35.564749 16255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:35.566205 16255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:35.566260 16255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:35.566289 16255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:35.567701 16255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:35.567999 16255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:35.568378 16255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:35.573557 16260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.573553 16261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.573553 16263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:35.573969 16255 server_base.cc:1034] running on GCE node
I20260928 12:58:35.574177 16255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:35.574429 16255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:35.575649 16255 hybrid_clock.cc:648] HybridClock initialized: now 1790600315575633 us; error 28 us; skew 500 ppm
I20260928 12:58:35.577064 16255 webserver.cc:460] Webserver started at http://127.11.11.67:32941/ using document root <none> and password file <none>
I20260928 12:58:35.577340 16255 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:35.577415 16255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:35.577581 16255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:35.578794 16255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "4129a301046c4c179fd82360c4cc795d"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "26dac37264db057b35baeaa9c976038c"
server_key_iv: "f91748d60c3dde726d0fab591a0b1db5"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.579174 16255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "4129a301046c4c179fd82360c4cc795d"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "26dac37264db057b35baeaa9c976038c"
server_key_iv: "f91748d60c3dde726d0fab591a0b1db5"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.580695 16255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:35.581430 16269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.581641 16255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:35.581748 16255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "4129a301046c4c179fd82360c4cc795d"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "26dac37264db057b35baeaa9c976038c"
server_key_iv: "f91748d60c3dde726d0fab591a0b1db5"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.581830 16255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:35.585417 16255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:35.585697 16255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:35.585839 16255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:35.586045 16255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:35.586349 16255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:35.586403 16255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.586443 16255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:35.586472 16255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.592394 16255 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40245
I20260928 12:58:35.592432 16382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40245 every 8 connection(s)
I20260928 12:58:35.592846 16255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:35.595875 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.593609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:39587 (local address 127.11.11.126:46401)
0928 12:58:35.593656 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.593658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.593899 (+ 241us) server_negotiation.cc:407] Connection header received
0928 12:58:35.594386 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.594386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.594391 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.594448 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.595691 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.595692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.595692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.595695 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.595706 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.595709 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.595763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.595808 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.595820 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:58:35.596169 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.593584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46401 (local address 127.11.11.67:39587)
0928 12:58:35.593767 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.593809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.594305 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.594517 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.594520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.594528 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.595542 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.595543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.595579 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.595611 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.595760 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.595763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.595765 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.595817 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.595865 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 12:58:35.596251 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16255
I20260928 12:58:35.596354 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:35.596549 11309 external_mini_cluster.cc:1428] Setting key 0cf0e9584ef12f511f90c083e35c29a6
I20260928 12:58:35.597051 16383 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46401
I20260928 12:58:35.597188 16383 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:35.597365 16383 heartbeater.cc:511] Master 127.11.11.126:46401 requested a full tablet report, sending...
I20260928 12:58:35.597836 15932 ts_manager.cc:194] Registered new tserver with Master: 4129a301046c4c179fd82360c4cc795d (127.11.11.67:40245)
I20260928 12:58:35.608881 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:58:35.611297 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.610319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.11.11.126:46401)
0928 12:58:35.610398 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.610400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.610452 (+ 52us) server_negotiation.cc:407] Connection header received
0928 12:58:35.610506 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.610507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.610511 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.610557 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.611055 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.611057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.611057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.611060 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.611079 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.611083 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.611131 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.611166 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.611179 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:58:35.612927 15932 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:35.614502 15932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:35.618516 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.616909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:40287 (local address 127.0.0.1:58650)
0928 12:58:35.617078 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.617121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.617182 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.617466 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.617473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.617480 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.618027 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.618027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.618043 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.618094 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.618331 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.618337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.618341 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.618416 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.618431 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20260928 12:58:35.618783 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.617105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.11.11.66:40287)
0928 12:58:35.617310 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.617317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.617328 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:35.617357 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.617359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.617370 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.617465 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.618189 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.618197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.618198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.618203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.618252 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.618278 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.618337 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.618657 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.618678 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260928 12:58:35.618950 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.617203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40245 (local address 127.0.0.1:52952)
0928 12:58:35.617865 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.617875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.617920 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.618110 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.618111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.618114 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.618558 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.618558 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.618573 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.618587 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.618823 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.618824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.618824 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.618865 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.618878 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":65,"threads_started":1}
I20260928 12:58:35.619251 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.616977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:41091 (local address 127.0.0.1:47834)
0928 12:58:35.617327 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.617337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.617383 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.617917 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.617918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.617921 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.618809 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.618809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.618816 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.618905 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.619133 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.619133 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.619134 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.619180 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.619192 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":122,"threads_started":1}
I20260928 12:58:35.619282 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.617078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.11.11.65:41091)
0928 12:58:35.617312 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.617318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.617693 (+ 375us) server_negotiation.cc:407] Connection header received
0928 12:58:35.617776 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.617782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.617791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.617881 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.618975 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.618983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.618983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.618987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.619025 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.619055 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.619124 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.619187 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.619203 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260928 12:58:35.619542 16186 tablet_service.cc:1467] Processing CreateTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb (DEFAULT_TABLE table=test-table [id=eeac6c467bd8419cbec18c282a9e4c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:35.619948 16186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fbdc2bba3ed422d94011e4f094ea3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:35.620157 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb (DEFAULT_TABLE table=test-table [id=eeac6c467bd8419cbec18c282a9e4c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:35.620479 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fbdc2bba3ed422d94011e4f094ea3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:35.622849 16397 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Bootstrap starting.
I20260928 12:58:35.623013 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.617404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.11.11.67:40245)
0928 12:58:35.617595 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.617600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.617976 (+ 376us) server_negotiation.cc:407] Connection header received
0928 12:58:35.618004 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.618007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.618017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.618091 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.618673 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.618679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.618679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.618683 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.618722 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.618748 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.618799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.622906 (+ 4107us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.622928 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260928 12:58:35.623396 16398 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: Bootstrap starting.
I20260928 12:58:35.623550 16397 tablet_bootstrap.cc:654] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:35.623781 16397 log.cc:824] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:35.623720 16317 tablet_service.cc:1467] Processing CreateTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb (DEFAULT_TABLE table=test-table [id=eeac6c467bd8419cbec18c282a9e4c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:35.624096 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fbdc2bba3ed422d94011e4f094ea3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:35.624095 16398 tablet_bootstrap.cc:654] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:35.624410 16398 log.cc:824] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:35.624487 16397 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: No bootstrap required, opened a new log
I20260928 12:58:35.624532 16397 ts_tablet_manager.cc:1397] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:35.625129 16398 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: No bootstrap required, opened a new log
I20260928 12:58:35.625185 16398 ts_tablet_manager.cc:1397] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:58:35.626752 16401 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Bootstrap starting.
I20260928 12:58:35.627224 16397 raft_consensus.cc:348] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.627434 16397 raft_consensus.cc:374] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:35.627475 16397 raft_consensus.cc:729] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763a47972e854f91b4d7cbb576c2b467, State: Initialized, Role: FOLLOWER
I20260928 12:58:35.627486 16401 tablet_bootstrap.cc:654] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:35.627619 16397 consensus_queue.cc:260] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.627796 16401 log.cc:824] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:35.627890 16397 ts_tablet_manager.cc:1428] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:58:35.627812 16398 raft_consensus.cc:348] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.627954 16398 raft_consensus.cc:374] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:35.628011 16252 heartbeater.cc:503] Master 127.11.11.126:46401 was elected leader, sending a full tablet report...
I20260928 12:58:35.628001 16398 raft_consensus.cc:729] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1319978e17cc4413bcc1a7f364bbc2ef, State: Initialized, Role: FOLLOWER
I20260928 12:58:35.628154 16398 consensus_queue.cc:260] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.628407 16398 ts_tablet_manager.cc:1428] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 12:58:35.628521 16121 heartbeater.cc:503] Master 127.11.11.126:46401 was elected leader, sending a full tablet report...
I20260928 12:58:35.628862 16401 tablet_bootstrap.cc:492] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: No bootstrap required, opened a new log
I20260928 12:58:35.628933 16401 ts_tablet_manager.cc:1397] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:35.630999 16401 raft_consensus.cc:348] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.631130 16401 raft_consensus.cc:374] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:35.631151 16401 raft_consensus.cc:729] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4129a301046c4c179fd82360c4cc795d, State: Initialized, Role: FOLLOWER
I20260928 12:58:35.631278 16401 consensus_queue.cc:260] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.631534 16401 ts_tablet_manager.cc:1428] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:35.631564 16383 heartbeater.cc:503] Master 127.11.11.126:46401 was elected leader, sending a full tablet report...
W20260928 12:58:35.728135 16253 tablet.cc:2369] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:35.728848 16405 raft_consensus.cc:482] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:35.728989 16405 raft_consensus.cc:504] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.729429 16405 leader_election.cc:288] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1319978e17cc4413bcc1a7f364bbc2ef (127.11.11.65:41091), 763a47972e854f91b4d7cbb576c2b467 (127.11.11.66:40287)
I20260928 12:58:35.730847 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.729644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:41091 (local address 127.11.11.67:36003)
0928 12:58:35.729702 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.729714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.729757 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.729908 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.729909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.729912 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.730416 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.730417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.730425 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.730440 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.730684 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.730685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.730686 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.730746 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.730762 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:58:35.730865 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.729722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:36003 (local address 127.11.11.65:41091)
0928 12:58:35.729766 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.729768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.729780 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:35.729812 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.729813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.729819 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.729899 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.730542 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.730543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.730544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.730563 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.730586 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.730592 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.730661 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.730760 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.730774 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:35.731009 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.729700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:60381 (local address 127.11.11.66:40287)
0928 12:58:35.729769 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.729772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.729917 (+ 145us) server_negotiation.cc:407] Connection header received
0928 12:58:35.729992 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.729993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.729997 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.730074 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.730742 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.730744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.730744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.730750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.730768 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.730773 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.730829 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.730914 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.730929 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:58:35.731197 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.729665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:40287 (local address 127.11.11.67:60381)
0928 12:58:35.729844 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.729852 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:35.729929 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:35.730078 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:35.730079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:35.730081 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:35.730608 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:58:35.730609 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:35.730615 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:58:35.730629 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:35.730844 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:35.730845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:35.730846 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:35.731048 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 12:58:35.731118 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260928 12:58:35.731192 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fbdc2bba3ed422d94011e4f094ea3fb" candidate_uuid: "4129a301046c4c179fd82360c4cc795d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" is_pre_election: true
I20260928 12:58:35.731374 16075 raft_consensus.cc:2454] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4129a301046c4c179fd82360c4cc795d in term 0.
I20260928 12:58:35.731504 16206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fbdc2bba3ed422d94011e4f094ea3fb" candidate_uuid: "4129a301046c4c179fd82360c4cc795d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763a47972e854f91b4d7cbb576c2b467" is_pre_election: true
I20260928 12:58:35.731637 16206 raft_consensus.cc:2454] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4129a301046c4c179fd82360c4cc795d in term 0.
I20260928 12:58:35.731628 16273 leader_election.cc:302] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1319978e17cc4413bcc1a7f364bbc2ef, 4129a301046c4c179fd82360c4cc795d; no voters:
I20260928 12:58:35.731746 16405 raft_consensus.cc:2781] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:35.731837 16405 raft_consensus.cc:482] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:35.731861 16405 raft_consensus.cc:3037] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.732851 16405 raft_consensus.cc:504] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.733001 16405 leader_election.cc:288] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [CANDIDATE]: Term 1 election: Requested vote from peers 1319978e17cc4413bcc1a7f364bbc2ef (127.11.11.65:41091), 763a47972e854f91b4d7cbb576c2b467 (127.11.11.66:40287)
I20260928 12:58:35.733194 16206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fbdc2bba3ed422d94011e4f094ea3fb" candidate_uuid: "4129a301046c4c179fd82360c4cc795d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763a47972e854f91b4d7cbb576c2b467"
I20260928 12:58:35.733191 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fbdc2bba3ed422d94011e4f094ea3fb" candidate_uuid: "4129a301046c4c179fd82360c4cc795d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1319978e17cc4413bcc1a7f364bbc2ef"
I20260928 12:58:35.733278 16206 raft_consensus.cc:3037] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.733278 16075 raft_consensus.cc:3037] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.734164 16206 raft_consensus.cc:2454] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4129a301046c4c179fd82360c4cc795d in term 1.
I20260928 12:58:35.734165 16075 raft_consensus.cc:2454] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4129a301046c4c179fd82360c4cc795d in term 1.
I20260928 12:58:35.734388 16273 leader_election.cc:302] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1319978e17cc4413bcc1a7f364bbc2ef, 4129a301046c4c179fd82360c4cc795d; no voters:
I20260928 12:58:35.734490 16405 raft_consensus.cc:2781] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:35.734629 16405 raft_consensus.cc:686] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 1 LEADER]: Becoming Leader. State: Replica: 4129a301046c4c179fd82360c4cc795d, State: Running, Role: LEADER
I20260928 12:58:35.734733 16405 consensus_queue.cc:237] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } }
I20260928 12:58:35.735608 15933 catalog_manager.cc:5495] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4129a301046c4c179fd82360c4cc795d (127.11.11.67). New cstate: current_term: 1 leader_uuid: "4129a301046c4c179fd82360c4cc795d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4129a301046c4c179fd82360c4cc795d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40245 } health_report { overall_health: HEALTHY } } }
I20260928 12:58:35.764401 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.763423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.11.11.67:40245)
0928 12:58:35.763489 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.763491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.763499 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:35.763549 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.763549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.763563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.763614 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.764214 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.764215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.764215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.764219 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.764236 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.764240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.764290 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.764334 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.764346 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:58:35.766507 16075 raft_consensus.cc:1261] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 1 FOLLOWER]: Refusing update from remote peer 4129a301046c4c179fd82360c4cc795d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:35.766507 16206 raft_consensus.cc:1261] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 1 FOLLOWER]: Refusing update from remote peer 4129a301046c4c179fd82360c4cc795d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:35.766925 16407 consensus_queue.cc:1035] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [LEADER]: Connected to new peer: Peer: permanent_uuid: "763a47972e854f91b4d7cbb576c2b467" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:35.767086 16405 consensus_queue.cc:1035] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1319978e17cc4413bcc1a7f364bbc2ef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:35.771129 16411 mvcc.cc:205] Tried to move back new op lower bound from 7334298893376344064 to 7334298893250256896. Current Snapshot: MvccSnapshot[applied={T|T < 7334298893376344064 or (T in {7334298893376344064})}]
I20260928 12:58:35.771947 15933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:35.772092 15933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:35.772907 15933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f32b88cbf11440fb84174002ec8b57c5: Sending DeleteTablet for 3 replicas of tablet 1fbdc2bba3ed422d94011e4f094ea3fb
I20260928 12:58:35.773217 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='slave'} at 127.0.0.1:47834
I20260928 12:58:35.773274 16317 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='slave'} at 127.0.0.1:52952
I20260928 12:58:35.773474 16420 tablet_replica.cc:330] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef: stopping tablet replica
I20260928 12:58:35.773510 16186 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fbdc2bba3ed422d94011e4f094ea3fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:35 UTC) from {username='slave'} at 127.0.0.1:58650
I20260928 12:58:35.773617 16420 raft_consensus.cc:2229] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:35.773746 16420 raft_consensus.cc:2258] T 1fbdc2bba3ed422d94011e4f094ea3fb P 1319978e17cc4413bcc1a7f364bbc2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:35.773766 16422 tablet_replica.cc:330] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: stopping tablet replica
I20260928 12:58:35.773910 16421 tablet_replica.cc:330] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: stopping tablet replica
I20260928 12:58:35.774014 16421 raft_consensus.cc:2229] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:35.774111 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15993
I20260928 12:58:35.774120 16421 raft_consensus.cc:2258] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:35.776142 16422 raft_consensus.cc:2229] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:35.776278 16422 raft_consensus.cc:2258] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:35.776621 16421 ts_tablet_manager.cc:1905] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:35.776676 16422 ts_tablet_manager.cc:1905] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:35.778507 16421 ts_tablet_manager.cc:1918] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:35.778599 16421 log.cc:1196] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/1fbdc2bba3ed422d94011e4f094ea3fb
I20260928 12:58:35.778672 16422 ts_tablet_manager.cc:1918] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:35.778736 16422 log.cc:1196] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/1fbdc2bba3ed422d94011e4f094ea3fb
I20260928 12:58:35.778885 16421 ts_tablet_manager.cc:1939] T 1fbdc2bba3ed422d94011e4f094ea3fb P 4129a301046c4c179fd82360c4cc795d: Deleting consensus metadata
I20260928 12:58:35.779003 16422 ts_tablet_manager.cc:1939] T 1fbdc2bba3ed422d94011e4f094ea3fb P 763a47972e854f91b4d7cbb576c2b467: Deleting consensus metadata
I20260928 12:58:35.779373 15920 catalog_manager.cc:4841] TS 4129a301046c4c179fd82360c4cc795d (127.11.11.67:40245): tablet 1fbdc2bba3ed422d94011e4f094ea3fb (table test-table [id=eeac6c467bd8419cbec18c282a9e4c77]) successfully deleted
I20260928 12:58:35.779681 15919 catalog_manager.cc:4841] TS 763a47972e854f91b4d7cbb576c2b467 (127.11.11.66:40287): tablet 1fbdc2bba3ed422d94011e4f094ea3fb (table test-table [id=eeac6c467bd8419cbec18c282a9e4c77]) successfully deleted
W20260928 12:58:35.781126 15920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:41091 (error 108)
I20260928 12:58:35.781292 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16124
I20260928 12:58:35.786357 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.782021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:41091 (local address 127.0.0.1:47844)
0928 12:58:35.786267 (+ 4246us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:35.786320 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:41091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4220}
W20260928 12:58:35.786573 15920 catalog_manager.cc:4584] TS 1319978e17cc4413bcc1a7f364bbc2ef (127.11.11.65:41091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fbdc2bba3ed422d94011e4f094ea3fb: Network error: Client connection negotiation failed: client connection to 127.11.11.65:41091: connect: Connection refused (error 111)
I20260928 12:58:35.787452 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16255
I20260928 12:58:35.792523 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 15902
2026-09-28T12:58:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (727 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 12:58:35.811666 11309 mini_oidc.cc:152] Starting JWKS server
I20260928 12:58:35.812844 11309 webserver.cc:460] Webserver started at https://127.0.0.1:34849/ using document root <none> and password file <none>
I20260928 12:58:35.812928 11309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:58:35.813059 11309 webserver.cc:460] Webserver started at http://127.0.0.1:40531/ using document root <none> and password file <none>
2026-09-28T12:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:35Z Disabled control of system clock
I20260928 12:58:35.821262 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35629
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:36733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:35629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/testchainca.cert with env {}
W20260928 12:58:35.904393 16429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:35.904618 16429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:35.904640 16429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:35.905808 16429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:35.905846 16429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:35.905859 16429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:35.905872 16429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:35.905884 16429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:58:35.905896 16429 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:58:35.905908 16429 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34849/jwks.json
I20260928 12:58:35.907414 16429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:35629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:35.907617 16429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:35.907785 16429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:35.912645 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.912750 16435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:35.912811 16434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:35.912657 16429 server_base.cc:1034] running on GCE node
I20260928 12:58:35.913204 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:35.913517 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:35.914698 16429 hybrid_clock.cc:648] HybridClock initialized: now 1790600315914678 us; error 52 us; skew 500 ppm
I20260928 12:58:35.916081 16429 webserver.cc:460] Webserver started at http://127.11.11.126:41951/ using document root <none> and password file <none>
I20260928 12:58:35.916353 16429 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:35.916427 16429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:35.916579 16429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:35.917670 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "3f9af5b7f88040caae2027d5363a95d3"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "305af7b69613e535f821ffc87ec62bc3"
server_key_iv: "99089de396c26914eb9dc9323ab560e4"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.918056 16429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "3f9af5b7f88040caae2027d5363a95d3"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "305af7b69613e535f821ffc87ec62bc3"
server_key_iv: "99089de396c26914eb9dc9323ab560e4"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.919421 16429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:35.920245 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:35.920423 16429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:35.920531 16429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "3f9af5b7f88040caae2027d5363a95d3"
format_stamp: "Formatted at 2026-09-28 12:58:35 on dist-test-slave-8sjz"
server_key: "305af7b69613e535f821ffc87ec62bc3"
server_key_iv: "99089de396c26914eb9dc9323ab560e4"
server_key_version: "encryptionkey@0"
I20260928 12:58:35.920609 16429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:35.932566 16429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:35.932924 16429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:35.933118 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:35.937328 16429 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:35629
I20260928 12:58:35.937362 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:35629 every 8 connection(s)
I20260928 12:58:35.937745 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:35.938710 16496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:35.942088 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3: Bootstrap starting.
I20260928 12:58:35.942844 16496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:35.943157 16496 log.cc:824] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:35.943953 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3: No bootstrap required, opened a new log
I20260928 12:58:35.945603 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16429
I20260928 12:58:35.945701 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:35.945641 16496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } }
I20260928 12:58:35.945899 16496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:35.945936 11309 external_mini_cluster.cc:1428] Setting key 1a70dd9cbc39cf1fd20bd5e254ec01e9
I20260928 12:58:35.945947 16496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9af5b7f88040caae2027d5363a95d3, State: Initialized, Role: FOLLOWER
I20260928 12:58:35.946127 16496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } }
I20260928 12:58:35.946205 16496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:35.946254 16496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:35.946303 16496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:35.947063 16496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } }
I20260928 12:58:35.947182 16496 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f9af5b7f88040caae2027d5363a95d3; no voters:
I20260928 12:58:35.947350 16496 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:35.947378 16501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:35.947484 16501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9af5b7f88040caae2027d5363a95d3, State: Running, Role: LEADER
I20260928 12:58:35.947567 16501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } }
I20260928 12:58:35.947629 16496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:35.947930 16502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9af5b7f88040caae2027d5363a95d3. Latest consensus state: current_term: 1 leader_uuid: "3f9af5b7f88040caae2027d5363a95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } } }
I20260928 12:58:35.947932 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f9af5b7f88040caae2027d5363a95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f9af5b7f88040caae2027d5363a95d3" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35629 } } }
I20260928 12:58:35.948160 16502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:35.948181 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:35.948518 16510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:35.949007 16510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:35.952272 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:35.946465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.11.11.126:35629)
0928 12:58:35.946697 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:35.946707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:35.946751 (+ 44us) server_negotiation.cc:407] Connection header received
0928 12:58:35.946875 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:35.946882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:35.946973 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:35.947147 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:35.947406 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.949409 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.950284 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:35.950697 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:35.950835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:35.951599 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:35.951603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:35.951604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:35.951778 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:35.951813 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:35.951824 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:35.951896 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:35.951986 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:58:35.952099 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260928 12:58:35.953220 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:58:35.956790 16510 catalog_manager.cc:1347] Generated new cluster ID: 66d0396402574ff6bb2bb2a706e473c8
I20260928 12:58:35.956861 16510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:35.985630 16510 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:35.986377 16510 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:36.004442 16510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9af5b7f88040caae2027d5363a95d3: Generated new TSK 0
I20260928 12:58:36.004673 16510 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:58:36.037508 16526 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 12:58:36.037627 16526 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:58:36.037904 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.021892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.11.11.126:35629)
0928 12:58:36.021960 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.021962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.022106 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:58:36.022216 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.022217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.022230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.022316 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:58:36.022479 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.022970 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.024281 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.024468 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.024546 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.024786 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:58:36.037642 (+ 12856us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:58:36.037834 (+ 192us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 12:58:36.037968 16500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 12:58:36.038295 16525 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:36.021831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35629 (local address 127.0.0.1:51158)
0928 12:58:36.022036 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:36.022046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:36.022129 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:36.022325 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:36.022325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:36.022327 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:58:36.022398 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.022400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.023122 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.023124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.024191 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.024194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.024565 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.024566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.024633 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.024659 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:58:36.037828 (+ 13169us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:58:36.037944 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.11.126:35629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.11.126:35629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 12:58:36.038815 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16429
2026-09-28T12:58:36Z chronyd exiting
I20260928 12:58:36.075167 11309 test_util.cc:175] -----------------------------------------------
I20260928 12:58:36.075256 11309 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600290352246-11309-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 12:58:36.075991 11309 mini_oidc.cc:152] Starting JWKS server
I20260928 12:58:36.076725 11309 webserver.cc:460] Webserver started at https://127.0.0.1:37999/ using document root <none> and password file <none>
I20260928 12:58:36.076769 11309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:58:36.076879 11309 webserver.cc:460] Webserver started at http://127.0.0.1:35905/ using document root <none> and password file <none>
2026-09-28T12:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:36Z Disabled control of system clock
I20260928 12:58:36.084728 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35957
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:35957
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37999/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/testchainca.cert with env {}
W20260928 12:58:36.165746 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:36.165964 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:36.165987 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:36.167238 16532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:36.167284 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:36.167296 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:36.167309 16532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:36.167321 16532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:58:36.167333 16532 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:58:36.167346 16532 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37999/jwks.json
I20260928 12:58:36.168889 16532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:35957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37999/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:36.169106 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:36.169279 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:36.174327 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:36.174355 16532 server_base.cc:1034] running on GCE node
W20260928 12:58:36.174317 16540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:36.174320 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:36.174791 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:36.175128 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:36.176293 16532 hybrid_clock.cc:648] HybridClock initialized: now 1790600316176282 us; error 41 us; skew 500 ppm
I20260928 12:58:36.177516 16532 webserver.cc:460] Webserver started at http://127.11.11.126:34421/ using document root <none> and password file <none>
I20260928 12:58:36.177791 16532 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:36.177863 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:36.178009 16532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:36.179088 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "21656fc726b94a7189941f2b70904daa"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "b43173b117f7ea6815e3efd7e25deeaf"
server_key_iv: "8d39caec730575a9cdf598c252dac334"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.179458 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "21656fc726b94a7189941f2b70904daa"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "b43173b117f7ea6815e3efd7e25deeaf"
server_key_iv: "8d39caec730575a9cdf598c252dac334"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.180912 16532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:36.181653 16546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:36.181845 16532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:36.181951 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "21656fc726b94a7189941f2b70904daa"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "b43173b117f7ea6815e3efd7e25deeaf"
server_key_iv: "8d39caec730575a9cdf598c252dac334"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.182029 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:36.216285 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:36.216647 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:36.216809 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:36.221339 16532 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:35957
I20260928 12:58:36.221370 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:35957 every 8 connection(s)
I20260928 12:58:36.221773 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:36.222467 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:36.225451 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa: Bootstrap starting.
I20260928 12:58:36.226073 16599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:36.226372 16599 log.cc:824] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:36.227136 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa: No bootstrap required, opened a new log
I20260928 12:58:36.228863 16599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } }
I20260928 12:58:36.229013 16599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:36.229036 16599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21656fc726b94a7189941f2b70904daa, State: Initialized, Role: FOLLOWER
I20260928 12:58:36.229233 16599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } }
I20260928 12:58:36.229314 16599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:36.229358 16599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:36.229409 16599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:36.229642 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16532
I20260928 12:58:36.229727 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:36.229905 11309 external_mini_cluster.cc:1428] Setting key 9e1b599b3dddc0423fc9c5fdc877c485
I20260928 12:58:36.230130 16599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } }
I20260928 12:58:36.230252 16599 leader_election.cc:302] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21656fc726b94a7189941f2b70904daa; no voters:
I20260928 12:58:36.230443 16599 leader_election.cc:288] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:36.230657 16604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:36.230729 16599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:36.230852 16604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [term 1 LEADER]: Becoming Leader. State: Replica: 21656fc726b94a7189941f2b70904daa, State: Running, Role: LEADER
I20260928 12:58:36.231010 16604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } }
I20260928 12:58:36.231447 16602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21656fc726b94a7189941f2b70904daa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } } }
I20260928 12:58:36.231539 16602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:36.231678 16606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21656fc726b94a7189941f2b70904daa. Latest consensus state: current_term: 1 leader_uuid: "21656fc726b94a7189941f2b70904daa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21656fc726b94a7189941f2b70904daa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35957 } } }
I20260928 12:58:36.231760 16606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:36.231913 16613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:36.232754 16613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:36.235010 16613 catalog_manager.cc:1347] Generated new cluster ID: f6aa54d7a0fe42e98fa4903c39fe5668
I20260928 12:58:36.235069 16613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:36.236409 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.230621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.11.11.126:35957)
0928 12:58:36.230833 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.230843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.230880 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:58:36.231015 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.231022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.231095 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.231256 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:36.231466 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.233122 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.234085 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.234661 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.234809 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.235557 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:36.235561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:36.235561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:36.235783 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:36.235828 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:36.235842 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:36.236083 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:36.236114 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:58:36.236195 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 12:58:36.237696 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:58:36.248803 16613 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:36.249568 16613 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:36.260687 16613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21656fc726b94a7189941f2b70904daa: Generated new TSK 0
I20260928 12:58:36.260944 16613 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:58:36.282652 16629 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 12:58:36.282763 16629 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:58:36.283128 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.268347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.11.11.126:35957)
0928 12:58:36.268397 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.268399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.268514 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:58:36.268581 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.268582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.268594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.268659 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:58:36.268817 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.269327 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.270507 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.270740 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.270796 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.270995 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:58:36.282928 (+ 11933us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:58:36.283071 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 12:58:36.283212 16605 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 12:58:36.283334 16628 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:36.268251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35957 (local address 127.0.0.1:35818)
0928 12:58:36.268407 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:36.268419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:36.268514 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:36.268679 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:36.268680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:36.268682 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:58:36.268747 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.268749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.269418 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.269419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.270395 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.270398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.270821 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.270822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.270885 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.270892 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:58:36.283066 (+ 12174us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:58:36.283228 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.11.126:35957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260928 12:58:36.283721 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16532
2026-09-28T12:58:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 12:58:36.321187 11309 mini_oidc.cc:152] Starting JWKS server
I20260928 12:58:36.321911 11309 webserver.cc:460] Webserver started at https://127.0.0.1:33349/ using document root <none> and password file <none>
I20260928 12:58:36.321981 11309 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:58:36.322144 11309 webserver.cc:460] Webserver started at http://127.0.0.1:33231/ using document root <none> and password file <none>
2026-09-28T12:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:36Z Disabled control of system clock
I20260928 12:58:36.330078 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45573
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45573
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33349/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:58:36.415622 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:36.415855 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:36.415880 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:36.417138 16635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:36.417200 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:36.417214 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:36.417227 16635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:36.417239 16635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:58:36.417260 16635 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:58:36.417277 16635 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33349/jwks.json
I20260928 12:58:36.418607 16635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33349/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:36.418824 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:36.419075 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:36.423905 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:36.423918 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:36.423918 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:36.424283 16635 server_base.cc:1034] running on GCE node
I20260928 12:58:36.424566 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:36.424866 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:36.426035 16635 hybrid_clock.cc:648] HybridClock initialized: now 1790600316426021 us; error 46 us; skew 500 ppm
I20260928 12:58:36.433380 16635 webserver.cc:460] Webserver started at http://127.11.11.126:36863/ using document root <none> and password file <none>
I20260928 12:58:36.433712 16635 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:36.433797 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:36.433964 16635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:36.435143 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "aa520c01ad8c4a5d928987c5cdfc04f6"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "ce6ee3b4c49f0a8e737adbd0a7cd8aac"
server_key_iv: "f2c9f4f80e4a8ffe6d58447f7474fd2d"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.435536 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "aa520c01ad8c4a5d928987c5cdfc04f6"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "ce6ee3b4c49f0a8e737adbd0a7cd8aac"
server_key_iv: "f2c9f4f80e4a8ffe6d58447f7474fd2d"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.437101 16635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:36.438047 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:36.438315 16635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:36.438434 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "aa520c01ad8c4a5d928987c5cdfc04f6"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "ce6ee3b4c49f0a8e737adbd0a7cd8aac"
server_key_iv: "f2c9f4f80e4a8ffe6d58447f7474fd2d"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.438514 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:36.447894 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:36.448277 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:36.448438 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:36.452605 16635 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45573
I20260928 12:58:36.452688 16701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45573 every 8 connection(s)
I20260928 12:58:36.453037 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:36.453696 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:36.454609 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16635
I20260928 12:58:36.454725 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:36.454922 11309 external_mini_cluster.cc:1428] Setting key e444c99eeeb520a45950f1fa8de7a086
I20260928 12:58:36.457588 16702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: Bootstrap starting.
I20260928 12:58:36.458297 16702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:36.458592 16702 log.cc:824] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:36.459314 16702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: No bootstrap required, opened a new log
I20260928 12:58:36.461287 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.455548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.11.11.126:45573)
0928 12:58:36.455778 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.455787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.455821 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:58:36.456425 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.456434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.456525 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.456696 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:36.456866 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.458205 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.459053 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.459501 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.459631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.460505 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:36.460515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:36.460515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:36.460745 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:36.460783 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:36.460795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:36.460881 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:36.461010 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 12:58:36.461103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 12:58:36.461424 16702 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } }
I20260928 12:58:36.461619 16702 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:36.461669 16702 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa520c01ad8c4a5d928987c5cdfc04f6, State: Initialized, Role: FOLLOWER
I20260928 12:58:36.461823 16702 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } }
I20260928 12:58:36.461927 16702 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:36.461964 16702 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:36.462010 16702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:36.462345 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:58:36.462736 16702 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } }
I20260928 12:58:36.462854 16702 leader_election.cc:302] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa520c01ad8c4a5d928987c5cdfc04f6; no voters:
I20260928 12:58:36.463027 16702 leader_election.cc:288] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:36.463182 16707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:36.463318 16702 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:36.463665 16707 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [term 1 LEADER]: Becoming Leader. State: Replica: aa520c01ad8c4a5d928987c5cdfc04f6, State: Running, Role: LEADER
I20260928 12:58:36.463795 16707 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } }
I20260928 12:58:36.464316 16709 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa520c01ad8c4a5d928987c5cdfc04f6. Latest consensus state: current_term: 1 leader_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } } }
I20260928 12:58:36.464423 16709 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:36.464586 16708 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa520c01ad8c4a5d928987c5cdfc04f6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45573 } } }
I20260928 12:58:36.464674 16708 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6 [sys.catalog]: This master's current role is: LEADER
W20260928 12:58:36.465508 16722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:36.465577 16722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:36.465646 16716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:36.465806 16716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:36.467409 16716 catalog_manager.cc:1347] Generated new cluster ID: 3fcddc2f0a9141068441b4fc476d5e21
I20260928 12:58:36.467490 16716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:36.477479 16716 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:36.478050 16716 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:36.485685 16716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa520c01ad8c4a5d928987c5cdfc04f6: Generated new TSK 0
I20260928 12:58:36.485952 16716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:36.502305 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.498427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.11.11.126:45573)
0928 12:58:36.498498 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.498500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.498631 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:58:36.498721 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.498721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.498732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.498801 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:58:36.498960 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.499473 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.500615 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.500822 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.500887 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.501097 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:58:36.502077 (+ 980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:58:36.502220 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 12:58:36.502373 16705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 12:58:36.502401 16731 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:36.498364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45573 (local address 127.0.0.1:50498)
0928 12:58:36.498548 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:36.498561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:36.498639 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:36.498821 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:36.498822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:36.498824 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:58:36.498895 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.498897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.499560 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.499561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.500526 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.500530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.500916 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.500917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.500985 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.500992 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:58:36.502184 (+ 1192us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:58:36.502296 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.11.126:45573: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33349/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260928 12:58:36.502663 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16635
E20260928 12:58:36.513131 16732 webserver.cc:533] Webserver: sslize failed:
E20260928 12:58:36.513226 16732 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T12:58:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T12:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:36Z Disabled control of system clock
I20260928 12:58:36.548270 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45635
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45635
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/testchainca.cert with env {}
W20260928 12:58:36.630712 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:36.630940 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:36.630964 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:36.632277 16736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:36.632323 16736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:36.632359 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:36.632372 16736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:36.632397 16736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:58:36.632409 16736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:58:36.632421 16736 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 12:58:36.633986 16736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:36.634238 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:36.634454 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:36.639398 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:36.639400 16742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:36.639463 16741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:36.639686 16736 server_base.cc:1034] running on GCE node
I20260928 12:58:36.639967 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:36.640308 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:36.641486 16736 hybrid_clock.cc:648] HybridClock initialized: now 1790600316641471 us; error 45 us; skew 500 ppm
I20260928 12:58:36.642866 16736 webserver.cc:460] Webserver started at http://127.11.11.126:46043/ using document root <none> and password file <none>
I20260928 12:58:36.643114 16736 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:36.643186 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:36.643344 16736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:36.644517 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "31bea262e05844d082c16f4c41423393"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "22727341075f55cb218f0a317c5fc09d"
server_key_iv: "50f9c517ddbda7a9afa65852e293a4ea"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.644965 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "31bea262e05844d082c16f4c41423393"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "22727341075f55cb218f0a317c5fc09d"
server_key_iv: "50f9c517ddbda7a9afa65852e293a4ea"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.646334 16736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:36.647059 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:36.647300 16736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:36.647392 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "31bea262e05844d082c16f4c41423393"
format_stamp: "Formatted at 2026-09-28 12:58:36 on dist-test-slave-8sjz"
server_key: "22727341075f55cb218f0a317c5fc09d"
server_key_iv: "50f9c517ddbda7a9afa65852e293a4ea"
server_key_version: "encryptionkey@0"
I20260928 12:58:36.647451 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:36.666810 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:36.667106 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:36.667212 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:36.671223 16736 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45635
I20260928 12:58:36.671265 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45635 every 8 connection(s)
I20260928 12:58:36.671653 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:36.672724 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16736
I20260928 12:58:36.672749 16803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:36.672806 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:36.673012 11309 external_mini_cluster.cc:1428] Setting key 0858596b2d757fe10ba5201b5675eab7
I20260928 12:58:36.676430 16803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393: Bootstrap starting.
I20260928 12:58:36.677141 16803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:36.677459 16803 log.cc:824] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:36.678326 16803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393: No bootstrap required, opened a new log
I20260928 12:58:36.679373 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.673582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.11.11.126:45635)
0928 12:58:36.673828 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.673838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.673873 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:58:36.674398 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.674408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.674498 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.674659 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:36.674878 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.676298 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.677127 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.677766 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.677865 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.678634 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:36.678639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:36.678639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:58:36.678859 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:36.678904 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:58:36.678920 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:36.678992 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:36.679058 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 12:58:36.679143 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260928 12:58:36.680548 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:58:36.680812 16803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } }
I20260928 12:58:36.680974 16803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:36.681020 16803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bea262e05844d082c16f4c41423393, State: Initialized, Role: FOLLOWER
I20260928 12:58:36.681144 16803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } }
I20260928 12:58:36.681236 16803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:36.681281 16803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:36.681331 16803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:36.682185 16803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } }
I20260928 12:58:36.682312 16803 leader_election.cc:302] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31bea262e05844d082c16f4c41423393; no voters:
I20260928 12:58:36.682504 16803 leader_election.cc:288] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:36.682552 16808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:36.682669 16808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [term 1 LEADER]: Becoming Leader. State: Replica: 31bea262e05844d082c16f4c41423393, State: Running, Role: LEADER
I20260928 12:58:36.682788 16803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:36.682770 16808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } }
I20260928 12:58:36.683223 16810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31bea262e05844d082c16f4c41423393. Latest consensus state: current_term: 1 leader_uuid: "31bea262e05844d082c16f4c41423393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } } }
I20260928 12:58:36.683257 16809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31bea262e05844d082c16f4c41423393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bea262e05844d082c16f4c41423393" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45635 } } }
I20260928 12:58:36.683379 16810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:36.683399 16809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:36.683878 16817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:36.684427 16817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:36.685994 16817 catalog_manager.cc:1347] Generated new cluster ID: db27a3b9fd8545f7919d8449832576a6
I20260928 12:58:36.686060 16817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:36.692489 16817 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:36.693029 16817 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:36.702073 16817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31bea262e05844d082c16f4c41423393: Generated new TSK 0
I20260928 12:58:36.702294 16817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:37.254684 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:36.712562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.11.11.126:45635)
0928 12:58:36.712618 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:36.712621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:36.712740 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:58:36.712829 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:36.712830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:36.712844 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:36.712931 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:58:36.713073 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.713597 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.714390 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.714594 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.714672 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.714863 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:58:37.254363 (+539500us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:58:37.254589 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 12:58:37.254765 16832 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:36.712495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45635 (local address 127.0.0.1:60364)
0928 12:58:36.712669 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:36.712679 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:36.712760 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:36.712931 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:36.712932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:36.712934 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:58:36.712996 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.712997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.713688 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.713689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.714299 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:36.714302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:36.714676 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:36.714677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:36.714737 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:36.714743 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:58:37.254530 (+539787us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:58:37.254663 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.11.126:45635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
W20260928 12:58:37.254793 16806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 12:58:37.255211 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16736
2026-09-28T12:58:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (732 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:37 dist-test-slave-8sjz krb5kdc[16844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:37 dist-test-slave-8sjz krb5kdc[16844](info): set up 2 sockets
Sep 28 12:58:37 dist-test-slave-8sjz krb5kdc[16844](info): commencing operation
krb5kdc: starting...
W20260928 12:58:38.469314 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:38 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:38.596138 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:33161
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:33161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:38.678771 16857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:38.679011 16857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:38.679036 16857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:38.680392 16857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:38.680444 16857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:38.680459 16857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:38.680471 16857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:38.680485 16857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:38.682034 16857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:33161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:33161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:38.682319 16857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:38.682581 16857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:38.688022 16863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:38.687994 16865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:38.688014 16862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:38.688542 16857 server_base.cc:1034] running on GCE node
I20260928 12:58:38.688834 16857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:38.689153 16857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:38.690327 16857 hybrid_clock.cc:648] HybridClock initialized: now 1790600318690299 us; error 42 us; skew 500 ppm
Sep 28 12:58:38 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600318, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:38.693041 16857 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:38.693557 16857 webserver.cc:460] Webserver started at http://127.11.11.126:38219/ using document root <none> and password file <none>
I20260928 12:58:38.693818 16857 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:38.693889 16857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:38.694039 16857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:38.695151 16857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "0552348dffbd4ba9b7ec65f4e549d6fa"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "a06b3df6001f5a894119e294f037f117"
server_key_iv: "2b371e9b6f6f7a0225712a4d5237d709"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.695528 16857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "0552348dffbd4ba9b7ec65f4e549d6fa"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "a06b3df6001f5a894119e294f037f117"
server_key_iv: "2b371e9b6f6f7a0225712a4d5237d709"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.697044 16857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:38.697871 16872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:38.698196 16857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:38.698321 16857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "0552348dffbd4ba9b7ec65f4e549d6fa"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "a06b3df6001f5a894119e294f037f117"
server_key_iv: "2b371e9b6f6f7a0225712a4d5237d709"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.698402 16857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:38.727311 16857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:38.729903 16857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:38.730087 16857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:38.734324 16857 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:33161
I20260928 12:58:38.734359 16924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:33161 every 8 connection(s)
I20260928 12:58:38.734742 16857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:38.735634 16925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:38.738148 16925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa: Bootstrap starting.
I20260928 12:58:38.738785 16925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:38.739095 16925 log.cc:824] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:38.739887 16925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa: No bootstrap required, opened a new log
I20260928 12:58:38.741748 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16857
I20260928 12:58:38.741858 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:38.741776 16925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } }
I20260928 12:58:38.741979 16925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:38.742018 16925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0552348dffbd4ba9b7ec65f4e549d6fa, State: Initialized, Role: FOLLOWER
I20260928 12:58:38.742053 11309 external_mini_cluster.cc:1428] Setting key 8a4117dc2a3570a36b33c8beda1ddb3d
I20260928 12:58:38.742182 16925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } }
I20260928 12:58:38.742275 16925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:38.742301 16925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:38.742375 16925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:38.743582 16925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } }
I20260928 12:58:38.743723 16925 leader_election.cc:302] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0552348dffbd4ba9b7ec65f4e549d6fa; no voters:
I20260928 12:58:38.743902 16925 leader_election.cc:288] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:38.743973 16930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:38.744143 16930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [term 1 LEADER]: Becoming Leader. State: Replica: 0552348dffbd4ba9b7ec65f4e549d6fa, State: Running, Role: LEADER
I20260928 12:58:38.744320 16925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:38.744297 16930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } }
I20260928 12:58:38.744828 16931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } } }
I20260928 12:58:38.744902 16931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:38.745098 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0552348dffbd4ba9b7ec65f4e549d6fa. Latest consensus state: current_term: 1 leader_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0552348dffbd4ba9b7ec65f4e549d6fa" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 33161 } } }
I20260928 12:58:38.745187 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:38.745607 16938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:38.746095 16938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:38.747848 16938 catalog_manager.cc:1347] Generated new cluster ID: 406e252e3f9f4d69aec225f42a05aa63
I20260928 12:58:38.747905 16938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:38 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:38.754575 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:38.742596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.11.11.126:33161)
0928 12:58:38.742811 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:38.742821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:38.742861 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:58:38.742969 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:38.742974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:38.743019 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:38.743132 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:38.743993 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.745454 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.746491 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.747105 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.747219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:38.750208 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:38.750213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:38.750213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:38.750226 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:38.752372 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:38.752836 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:38.752839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:38.752839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:38.752977 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:38.753196 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:38.753196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:38.753197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:38.753530 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:38.754115 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:38.754193 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:58:38.754347 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":131,"threads_started":1}
I20260928 12:58:38.754624 16938 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:38.755657 16938 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:38.763196 16938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0552348dffbd4ba9b7ec65f4e549d6fa: Generated new TSK 0
I20260928 12:58:38.763407 16938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:38.794764 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:38.878676 16951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:38.878942 16951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:38.878985 16951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:38.880239 16951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:38.880313 16951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:38.880360 16951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:38.881919 16951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:38.882198 16951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:38.882443 16951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:38.887691 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:38.887811 16951 server_base.cc:1034] running on GCE node
W20260928 12:58:38.887682 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:38.887682 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:38.888195 16951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:38.888440 16951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:38.889595 16951 hybrid_clock.cc:648] HybridClock initialized: now 1790600318889578 us; error 31 us; skew 500 ppm
Sep 28 12:58:38 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600318, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:38.892326 16951 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:38.892839 16951 webserver.cc:460] Webserver started at http://127.11.11.65:39723/ using document root <none> and password file <none>
I20260928 12:58:38.893078 16951 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:38.893147 16951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:38.893311 16951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:38.894405 16951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "90d4684a22ac4e8f94629a500ee2311c"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "9ed9c53be880cc1dfc490c54849eff1e"
server_key_iv: "85ac98ef768523e8cc009e8c3d625e8d"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.894785 16951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "90d4684a22ac4e8f94629a500ee2311c"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "9ed9c53be880cc1dfc490c54849eff1e"
server_key_iv: "85ac98ef768523e8cc009e8c3d625e8d"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.896194 16951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:38.896962 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:38.897130 16951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:38.897248 16951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "90d4684a22ac4e8f94629a500ee2311c"
format_stamp: "Formatted at 2026-09-28 12:58:38 on dist-test-slave-8sjz"
server_key: "9ed9c53be880cc1dfc490c54849eff1e"
server_key_iv: "85ac98ef768523e8cc009e8c3d625e8d"
server_key_version: "encryptionkey@0"
I20260928 12:58:38.897324 16951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:38.908457 16951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:38.911003 16951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:38.911185 16951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:38.911450 16951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:38.911782 16951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:38.911839 16951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:38.911881 16951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:38.911924 16951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:38.917847 16951 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:43975
I20260928 12:58:38.917891 17079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:43975 every 8 connection(s)
I20260928 12:58:38.918241 16951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:38.920377 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 16951
I20260928 12:58:38.920504 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:38.920706 11309 external_mini_cluster.cc:1428] Setting key b4f3ef11c2aae637d663267eaeb4d534
Sep 28 12:58:38 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600318, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 12:58:38.929114 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:38.919013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:32845 (local address 127.11.11.126:33161)
0928 12:58:38.919075 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:38.919077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:38.919301 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:58:38.920309 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:38.920310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:38.920326 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:38.920390 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:38.921302 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.921934 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.923326 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.923564 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.923657 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:38.926008 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:38.926010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:38.926011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:38.926023 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:38.927539 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:38.927993 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:38.927994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:38.927995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:38.928113 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:38.928404 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:38.928405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:38.928405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:38.928536 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:38.928606 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:38.928918 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 12:58:38.929017 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:58:38.929353 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:38.918953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:33161 (local address 127.11.11.65:32845)
0928 12:58:38.919117 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:38.919162 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:38.920166 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:38.920513 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:38.920519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:38.920917 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:38.921193 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:38.921198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.922041 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.922042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:38.923224 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:38.923227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:38.923671 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:38.923672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:38.923805 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:38.924412 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:38.924421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:38.925900 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:38.927650 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:38.927659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:38.927662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:38.927909 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:38.928189 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:38.928189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:38.928190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:38.928324 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:38.928607 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:38.928616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:38.928773 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 12:58:38.928883 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:58:38.929063 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 12:58:38.929960 17080 heartbeater.cc:347] Connected to a master server at 127.11.11.126:33161
I20260928 12:58:38.930110 17080 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:38.930444 17080 heartbeater.cc:511] Master 127.11.11.126:33161 requested a full tablet report, sending...
I20260928 12:58:38.931083 16889 ts_manager.cc:194] Registered new tserver with Master: 90d4684a22ac4e8f94629a500ee2311c (127.11.11.65:43975)
I20260928 12:58:38.931744 16889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:32845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:38.947690 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:39.034740 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:39.035033 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:39.035072 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:39.036720 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:39.036780 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:39.036813 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:39.038429 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:39.038640 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:39.038837 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:39.043936 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:39.043926 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:39.044131 17085 server_base.cc:1034] running on GCE node
W20260928 12:58:39.043982 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:39.044543 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:39.044817 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:39.045992 17085 hybrid_clock.cc:648] HybridClock initialized: now 1790600319045977 us; error 30 us; skew 500 ppm
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.048597 17085 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:39.049032 17085 webserver.cc:460] Webserver started at http://127.11.11.66:39715/ using document root <none> and password file <none>
I20260928 12:58:39.049225 17085 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:39.049269 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:39.049365 17085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:39.050419 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "b28cac11d8e04a5284b9d9b65a8810c4"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "195733c26c24b1b745f5935a4044dbdc"
server_key_iv: "0d2956f54867fc15ccd87e5368c3a083"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.050732 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "b28cac11d8e04a5284b9d9b65a8810c4"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "195733c26c24b1b745f5935a4044dbdc"
server_key_iv: "0d2956f54867fc15ccd87e5368c3a083"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.052182 17085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:39.053076 17100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.053332 17085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:39.053413 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "b28cac11d8e04a5284b9d9b65a8810c4"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "195733c26c24b1b745f5935a4044dbdc"
server_key_iv: "0d2956f54867fc15ccd87e5368c3a083"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.053504 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:39.075724 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:39.078245 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:39.078404 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:39.078685 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:39.079032 17085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:39.079067 17085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.079110 17085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:39.079138 17085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.085290 17085 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:34597
I20260928 12:58:39.085367 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:34597 every 8 connection(s)
I20260928 12:58:39.085731 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:39.093109 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17085
I20260928 12:58:39.093199 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:39.093493 11309 external_mini_cluster.cc:1428] Setting key 337d19e8460e9b9d6fdfb9706a6ef1f6
I20260928 12:58:39.096279 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.086552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:48313 (local address 127.11.11.126:33161)
0928 12:58:39.086601 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.086604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.086820 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:58:39.087876 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.087877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.087892 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.087945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.088684 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.089199 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.090247 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.090478 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.090560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.092958 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:39.092960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:39.092960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:39.092972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:39.094749 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.095213 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.095214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.095215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.095279 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.095544 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.095545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.095545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.095683 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:39.095758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:39.096091 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 12:58:39.096188 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:39.096490 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.086518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:33161 (local address 127.11.11.66:48313)
0928 12:58:39.086669 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:39.086724 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:39.087744 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:39.088019 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:39.088023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:39.088363 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:39.088565 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:39.088570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.089305 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.089306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:39.090148 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:39.090150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.090566 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.090567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:39.090711 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.091236 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:39.091246 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:39.092848 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:39.094858 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:39.094869 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:39.094871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:39.095133 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:39.095349 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:39.095350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:39.095351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:39.095474 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:39.095759 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:39.095767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:39.095920 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 12:58:39.096028 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:58:39.096230 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 12:58:39.097036 17214 heartbeater.cc:347] Connected to a master server at 127.11.11.126:33161
I20260928 12:58:39.097151 17214 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:39.097347 17214 heartbeater.cc:511] Master 127.11.11.126:33161 requested a full tablet report, sending...
I20260928 12:58:39.097805 16889 ts_manager.cc:194] Registered new tserver with Master: b28cac11d8e04a5284b9d9b65a8810c4 (127.11.11.66:34597)
I20260928 12:58:39.098364 16889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:48313
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:39.122539 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:39.205572 17219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:39.205811 17219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:39.205837 17219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:39.206995 17219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:39.207033 17219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:39.207060 17219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:39.208643 17219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:39.208931 17219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:39.209167 17219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:39.214408 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:39.214408 17227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:39.214417 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:39.214758 17219 server_base.cc:1034] running on GCE node
I20260928 12:58:39.214987 17219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:39.215241 17219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:39.216406 17219 hybrid_clock.cc:648] HybridClock initialized: now 1790600319216393 us; error 41 us; skew 500 ppm
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.219106 17219 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:39.219604 17219 webserver.cc:460] Webserver started at http://127.11.11.67:45473/ using document root <none> and password file <none>
I20260928 12:58:39.219868 17219 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:39.219918 17219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:39.220108 17219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:39.221252 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "4c895b3cc52a4c5ebb9c6d0d895bf603"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "d7fd32ce6d034af2b15e421b5f066da9"
server_key_iv: "80e13851a22af3676b3f9f0574c916f7"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.221634 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "4c895b3cc52a4c5ebb9c6d0d895bf603"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "d7fd32ce6d034af2b15e421b5f066da9"
server_key_iv: "80e13851a22af3676b3f9f0574c916f7"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.223070 17219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:39.223928 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.224186 17219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:39.224289 17219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "4c895b3cc52a4c5ebb9c6d0d895bf603"
format_stamp: "Formatted at 2026-09-28 12:58:39 on dist-test-slave-8sjz"
server_key: "d7fd32ce6d034af2b15e421b5f066da9"
server_key_iv: "80e13851a22af3676b3f9f0574c916f7"
server_key_version: "encryptionkey@0"
I20260928 12:58:39.224354 17219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:39.234887 17219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:39.237454 17219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:39.237627 17219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:39.237864 17219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:39.238134 17219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:39.238164 17219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.238188 17219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:39.238200 17219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:39.244277 17219 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35221
I20260928 12:58:39.244351 17347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35221 every 8 connection(s)
I20260928 12:58:39.244717 17219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:39.247905 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17219
I20260928 12:58:39.248006 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:39.248237 11309 external_mini_cluster.cc:1428] Setting key fdd718e4472960d89b746831752c4783
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:39.255505 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.245581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:38923 (local address 127.11.11.126:33161)
0928 12:58:39.245657 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.245660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.245973 (+ 313us) server_negotiation.cc:407] Connection header received
0928 12:58:39.246870 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.246872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.246883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.246938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.247542 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.248330 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.249685 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.249924 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.249989 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.252356 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:39.252358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:39.252358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:39.252365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:39.253928 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.254581 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.254582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.254582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.254641 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.254883 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.254884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.254884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.254995 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:39.255049 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:39.255363 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 12:58:39.255433 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:58:39.255803 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.245570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:33161 (local address 127.11.11.67:38923)
0928 12:58:39.245774 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:39.245838 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:39.246775 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:39.246998 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:39.247001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:39.247275 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:39.247472 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:39.247476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.248454 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.248455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:39.249580 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:39.249583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.250030 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.250031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:39.250146 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.250746 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:39.250754 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:39.252258 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:39.254068 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:39.254077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:39.254080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:39.254502 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:39.254707 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:39.254708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:39.254708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:39.254827 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:39.255077 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:39.255083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:39.255236 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 12:58:39.255336 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:58:39.255472 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260928 12:58:39.256510 17348 heartbeater.cc:347] Connected to a master server at 127.11.11.126:33161
I20260928 12:58:39.256678 17348 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:39.256883 17348 heartbeater.cc:511] Master 127.11.11.126:33161 requested a full tablet report, sending...
I20260928 12:58:39.257284 16889 ts_manager.cc:194] Registered new tserver with Master: 4c895b3cc52a4c5ebb9c6d0d895bf603 (127.11.11.67:35221)
I20260928 12:58:39.257721 16889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:38923
I20260928 12:58:39.259944 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/test-admin.keytab.
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.277371 11309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 12:58:39.279340 17355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:39.278449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.11.11.65:43975)
0928 12:58:39.278617 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.278622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.278630 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:39.278656 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.278658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.278671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.278759 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.279230 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48098: BlockingRecv error: recv got EOF from 127.0.0.1:48098 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
W20260928 12:58:39.279443 11309 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.280527 11309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.281491 11309 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:39.289788 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.282029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.11.11.65:43975)
0928 12:58:39.282087 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.282089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.282153 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:58:39.282236 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.282237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.282249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.282322 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.282734 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.283472 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.284340 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.284685 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.284781 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.287020 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:39.287030 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:39.287031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:39.287045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:39.288564 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.288932 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.288935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.288936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.288990 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.289186 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.289187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.289187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.289331 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:39.289505 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:39.289610 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 12:58:39.289695 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 12:58:39.291723 17355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:39.291097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.11.11.65:43975)
0928 12:58:39.291152 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.291154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.291164 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:39.291256 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.291257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.291269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.291324 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.291672 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48114: BlockingRecv error: recv got EOF from 127.0.0.1:48114 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 12:58:39.291810 11309 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.292871 11309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.293781 11309 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:39.301033 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.294345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.11.11.65:43975)
0928 12:58:39.294392 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.294394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.294403 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:39.294451 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.294452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.294459 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.294508 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.294854 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.295345 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.296000 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.296302 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.296360 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.298435 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:39.298437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:39.298437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:39.298445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:39.299796 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.300201 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.300202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.300203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.300264 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.300466 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.300467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.300468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.300596 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:39.300687 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:39.300818 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:58:39.300904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 12:58:39.302561 17355 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:39.301980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.11.11.65:43975)
0928 12:58:39.302014 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.302016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.302129 (+ 113us) server_negotiation.cc:407] Connection header received
0928 12:58:39.302198 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.302199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.302226 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.302293 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.302528 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48120: BlockingRecv error: recv got EOF from 127.0.0.1:48120 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:58:39.302651 11309 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600290352246-11309-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.303706 11309 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:39.304688 11309 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[16844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 12:58:39.311503 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:39.305206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.11.11.65:43975)
0928 12:58:39.305260 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:39.305262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:39.305271 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:39.305325 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:39.305326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:39.305335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:39.305401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:39.305779 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.306235 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.306937 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:39.307120 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:39.307202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:39.309343 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:39.309345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:39.309345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:39.309351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:39.310551 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.310864 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.310864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.310864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.310925 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:39.311090 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:39.311091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:39.311091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:39.311175 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:39.311253 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:39.311333 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 12:58:39.311387 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:39.312227 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16951
I20260928 12:58:39.317269 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17085
I20260928 12:58:39.321769 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17219
I20260928 12:58:39.326697 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 16857
2026-09-28T12:58:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1892 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[17368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[17368](info): set up 2 sockets
Sep 28 12:58:39 dist-test-slave-8sjz krb5kdc[17368](info): commencing operation
krb5kdc: starting...
W20260928 12:58:40.361235 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab.
Entry for principal oryx/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab.
I20260928 12:58:40.483284 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42529
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab
--principal=oryx/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:40.570214 17381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:40.570467 17381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:40.570508 17381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:40.571710 17381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:40.571781 17381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:40.571805 17381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:40.571835 17381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:40.571878 17381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:40.573453 17381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:42529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab
--principal=oryx/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:40.573751 17381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:40.573982 17381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:40.578896 17387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.578889 17386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.578893 17389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:40.579221 17381 server_base.cc:1034] running on GCE node
I20260928 12:58:40.579445 17381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:40.579706 17381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:40.580848 17381 hybrid_clock.cc:648] HybridClock initialized: now 1790600320580830 us; error 32 us; skew 500 ppm
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:40.583328 17381 init.cc:377] Logged in from keytab as oryx/127.11.11.126@KRBTEST.COM (short username oryx)
I20260928 12:58:40.583837 17381 webserver.cc:460] Webserver started at http://127.11.11.126:37131/ using document root <none> and password file <none>
I20260928 12:58:40.584138 17381 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:40.584205 17381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:40.584340 17381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:40.585428 17381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "ea609adf9d624c838c6baf7683f4e674"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "8e03678bdc13571221172ad3f40672fc"
server_key_iv: "4155ddd5a7393156957b77845cfd58a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.585804 17381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "ea609adf9d624c838c6baf7683f4e674"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "8e03678bdc13571221172ad3f40672fc"
server_key_iv: "4155ddd5a7393156957b77845cfd58a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.587206 17381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:40.587952 17396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.588146 17381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:40.588263 17381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "ea609adf9d624c838c6baf7683f4e674"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "8e03678bdc13571221172ad3f40672fc"
server_key_iv: "4155ddd5a7393156957b77845cfd58a8"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.588338 17381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:40.606878 17381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:40.609157 17381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:40.609334 17381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:40.613852 17381 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:42529
I20260928 12:58:40.613883 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:42529 every 8 connection(s)
I20260928 12:58:40.614275 17381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:40.615324 17449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:40.617762 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Bootstrap starting.
I20260928 12:58:40.618366 17449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:40.618661 17449 log.cc:824] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:40.618664 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17381
I20260928 12:58:40.618788 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:40.618993 11309 external_mini_cluster.cc:1428] Setting key a4294da1f6397d380b3d00f9de2c58d6
I20260928 12:58:40.619527 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: No bootstrap required, opened a new log
I20260928 12:58:40.621814 17449 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } }
I20260928 12:58:40.621976 17449 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:40.622022 17449 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea609adf9d624c838c6baf7683f4e674, State: Initialized, Role: FOLLOWER
I20260928 12:58:40.622193 17449 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } }
I20260928 12:58:40.622282 17449 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:40.622328 17449 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:40.622375 17449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:40.622928 17449 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } }
I20260928 12:58:40.623045 17449 leader_election.cc:302] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea609adf9d624c838c6baf7683f4e674; no voters:
I20260928 12:58:40.623242 17449 leader_election.cc:288] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:40.623297 17454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:40.623451 17454 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 LEADER]: Becoming Leader. State: Replica: ea609adf9d624c838c6baf7683f4e674, State: Running, Role: LEADER
I20260928 12:58:40.623540 17449 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:40.623558 17454 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } }
I20260928 12:58:40.623867 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea609adf9d624c838c6baf7683f4e674. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } }
I20260928 12:58:40.623987 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:40.623878 17456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } }
I20260928 12:58:40.624167 17456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:40.624320 17460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:40.624877 17460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:40.626566 17460 catalog_manager.cc:1347] Generated new cluster ID: dbe20a57464a422e89bde1a8a8d26844
I20260928 12:58:40.626617 17460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:40.631974 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:40.619554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.11.11.126:42529)
0928 12:58:40.619753 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:40.619760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:40.619786 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:58:40.620769 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:40.620778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:40.620831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:40.620954 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:40.621759 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.623048 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.624244 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.624982 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.625081 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:40.628069 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:40.628073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:40.628074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:40.628087 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:40.630118 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.630507 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.630511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.630511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.630627 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.630830 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.630831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.630831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.631177 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:40.631337 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:40.631452 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:58:40.631600 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260928 12:58:40.636242 17460 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:40.636842 17460 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:40.639878 17460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Generated new TSK 0
I20260928 12:58:40.640100 17460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
I20260928 12:58:40.669111 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:40.754161 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:40.754436 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:40.754480 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:40.755726 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:40.755796 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:40.755846 17475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:40.757447 17475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:40.757733 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:40.757977 17475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:40.763338 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.763324 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.763324 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:40.763358 17475 server_base.cc:1034] running on GCE node
I20260928 12:58:40.763798 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:40.764093 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:40.765251 17475 hybrid_clock.cc:648] HybridClock initialized: now 1790600320765239 us; error 28 us; skew 500 ppm
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:40.767882 17475 init.cc:377] Logged in from keytab as oryx/127.11.11.65@KRBTEST.COM (short username oryx)
I20260928 12:58:40.768409 17475 webserver.cc:460] Webserver started at http://127.11.11.65:41395/ using document root <none> and password file <none>
I20260928 12:58:40.768639 17475 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:40.768713 17475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:40.768862 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:40.769968 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "e42cc89910ff2ddb800f0aa7284530b9"
server_key_iv: "b7b4f01c1ea949755cffed099e677c52"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.770340 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "e42cc89910ff2ddb800f0aa7284530b9"
server_key_iv: "b7b4f01c1ea949755cffed099e677c52"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.771759 17475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:40.772579 17490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.772748 17475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.772851 17475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "e42cc89910ff2ddb800f0aa7284530b9"
server_key_iv: "b7b4f01c1ea949755cffed099e677c52"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.772935 17475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:40.786379 17475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:40.788566 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:40.788746 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:40.789001 17475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:40.789369 17475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:40.789423 17475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.789461 17475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:40.789501 17475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.795614 17475 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:36417
I20260928 12:58:40.795660 17603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:36417 every 8 connection(s)
I20260928 12:58:40.796003 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:40.804791 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17475
I20260928 12:58:40.804908 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:40.805117 11309 external_mini_cluster.cc:1428] Setting key ce06e2b33ad507f1aa25208d026f1a93
I20260928 12:58:40.805274 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:40.796778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:40121 (local address 127.11.11.126:42529)
0928 12:58:40.796825 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:40.796827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:40.797044 (+ 217us) server_negotiation.cc:407] Connection header received
0928 12:58:40.797537 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:40.797538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:40.797548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:40.797602 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:40.798632 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.799145 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.800255 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.800460 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.800516 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:40.802619 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:40.802620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:40.802621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:40.802628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:40.803847 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.804269 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.804270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.804271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.804325 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.804525 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.804526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.804526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.804646 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:40.804692 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:40.805074 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 12:58:40.805148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:40.805533 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:40.796765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:40121)
0928 12:58:40.796920 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:40.796958 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:40.797451 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:40.797660 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:40.797663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:40.798277 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:40.798531 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:40.798538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.799258 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.799258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:40.800169 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:40.800172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.800536 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.800537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:40.800637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:40.801151 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:40.801160 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:40.802520 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:40.803917 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:40.803924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:40.803926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:40.804202 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:40.804379 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:40.804380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:40.804380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:40.804486 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:40.804727 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:40.804735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:40.804924 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 12:58:40.805060 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 12:58:40.805244 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 12:58:40.806145 17604 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:40.806254 17604 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:40.806461 17604 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:40.806954 17413 ts_manager.cc:194] Registered new tserver with Master: bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:40.807928 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:40121
WARNING: no policy specified for oryx/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
I20260928 12:58:40.831439 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:40.913333 17609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:40.913558 17609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:40.913583 17609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:40.914726 17609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:40.914767 17609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:40.914793 17609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:40.916230 17609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:40.916426 17609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:40.916610 17609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:40.921618 17617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.921624 17615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:40.921624 17614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:40.921705 17609 server_base.cc:1034] running on GCE node
I20260928 12:58:40.922107 17609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:40.922349 17609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:40.923558 17609 hybrid_clock.cc:648] HybridClock initialized: now 1790600320923537 us; error 29 us; skew 500 ppm
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:40.926271 17609 init.cc:377] Logged in from keytab as oryx/127.11.11.66@KRBTEST.COM (short username oryx)
I20260928 12:58:40.926678 17609 webserver.cc:460] Webserver started at http://127.11.11.66:45211/ using document root <none> and password file <none>
I20260928 12:58:40.926877 17609 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:40.926920 17609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:40.927013 17609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:40.928228 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "253de7da96bc49d78de9c9e49bb55a2c"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "85cc5f5e4a753f3fee583f56dc1cbee8"
server_key_iv: "b4f9636506ab6786f4140476ba646b29"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.928555 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "253de7da96bc49d78de9c9e49bb55a2c"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "85cc5f5e4a753f3fee583f56dc1cbee8"
server_key_iv: "b4f9636506ab6786f4140476ba646b29"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.929899 17609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:58:40.930630 17624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.930807 17609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:40.930912 17609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "253de7da96bc49d78de9c9e49bb55a2c"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "85cc5f5e4a753f3fee583f56dc1cbee8"
server_key_iv: "b4f9636506ab6786f4140476ba646b29"
server_key_version: "encryptionkey@0"
I20260928 12:58:40.930997 17609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:40.944532 17609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:40.946570 17609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:40.946748 17609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:40.947002 17609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:40.947325 17609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:40.947378 17609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.947420 17609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:40.947451 17609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:40.953518 17609 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:39295
I20260928 12:58:40.953560 17737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:39295 every 8 connection(s)
I20260928 12:58:40.953909 17609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:40.956296 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17609
I20260928 12:58:40.956382 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:40.956571 11309 external_mini_cluster.cc:1428] Setting key afe67574605f1515c472157cf63694c2
Sep 28 12:58:40 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:40.965610 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:40.954672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:45221 (local address 127.11.11.126:42529)
0928 12:58:40.954730 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:40.954732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:40.954957 (+ 225us) server_negotiation.cc:407] Connection header received
0928 12:58:40.955435 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:40.955436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:40.955448 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:40.955501 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:40.956630 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.957252 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.958891 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.959178 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.959227 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:40.962447 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:40.962448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:40.962449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:40.962458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:40.963963 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.964534 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.964535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.964536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.964592 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:40.964905 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:40.964906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:40.964906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:40.965017 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:40.965066 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:40.965434 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 12:58:40.965538 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:58:40.965919 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:40.954640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:45221)
0928 12:58:40.954830 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:40.954867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:40.955351 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:40.955557 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:40.955561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:40.956258 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:40.956532 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:40.956538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.957356 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.957357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:40.958764 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:40.958767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:40.959335 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:40.959336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:40.959492 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:40.960213 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:40.960225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:40.962268 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:40.964085 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:40.964093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:40.964096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:40.964451 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:40.964663 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:40.964665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:40.964665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:40.964834 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:40.965079 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:40.965086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:40.965275 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 12:58:40.965406 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 12:58:40.965605 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for oryx/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 12:58:40.966467 17738 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:40.966563 17738 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:40.966780 17738 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:40.967196 17413 ts_manager.cc:194] Registered new tserver with Master: 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66:39295)
I20260928 12:58:40.967600 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:45221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
I20260928 12:58:40.983114 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:41.068766 17743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:41.068980 17743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:41.069006 17743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:41.070132 17743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:41.070169 17743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:41.070195 17743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:41.071585 17743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:42529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:41.071792 17743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:41.071978 17743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.076951 17748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.076959 17749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.077303 17743 server_base.cc:1034] running on GCE node
W20260928 12:58:41.076951 17751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.077661 17743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.077934 17743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.079090 17743 hybrid_clock.cc:648] HybridClock initialized: now 1790600321079085 us; error 33 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.081563 17743 init.cc:377] Logged in from keytab as oryx/127.11.11.67@KRBTEST.COM (short username oryx)
I20260928 12:58:41.082036 17743 webserver.cc:460] Webserver started at http://127.11.11.67:34271/ using document root <none> and password file <none>
I20260928 12:58:41.082245 17743 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.082286 17743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.082381 17743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:41.083410 17743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "3a37118cdffd4b44af5bf4968552ec88"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "1ba28c01eaa6214c63ff65ac94e637c0"
server_key_iv: "b517c5f6a57b5b98f18657533d4e9877"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.083735 17743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "3a37118cdffd4b44af5bf4968552ec88"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "1ba28c01eaa6214c63ff65ac94e637c0"
server_key_iv: "b517c5f6a57b5b98f18657533d4e9877"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.085078 17743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:41.085786 17758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.085934 17743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.086019 17743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "3a37118cdffd4b44af5bf4968552ec88"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "1ba28c01eaa6214c63ff65ac94e637c0"
server_key_iv: "b517c5f6a57b5b98f18657533d4e9877"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.086076 17743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.094909 17743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.097407 17743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:41.097533 17743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.097749 17743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:41.098075 17743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:41.098109 17743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.098132 17743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:41.098172 17743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.103943 17743 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40209
I20260928 12:58:41.104007 17871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40209 every 8 connection(s)
I20260928 12:58:41.104439 17743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:41.108325 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17743
I20260928 12:58:41.108436 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:41.108645 11309 external_mini_cluster.cc:1428] Setting key 3188a62bc08c0b6649d54f86becc1dea
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:41.114365 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.105283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:58363 (local address 127.11.11.126:42529)
0928 12:58:41.105346 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.105348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.105530 (+ 182us) server_negotiation.cc:407] Connection header received
0928 12:58:41.106030 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.106031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.106041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.106113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.107101 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.107679 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.109027 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.109261 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.109339 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.111578 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.111579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.111580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.111587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.113105 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.113470 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.113470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.113471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.113520 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.113763 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.113765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.113765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.113913 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.113983 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.114228 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.114296 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:58:41.114590 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.105234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:58363)
0928 12:58:41.105394 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.105438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.105938 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.106173 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.106177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.106773 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.106978 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.106983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.107770 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.107771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.108850 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.108853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.109363 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.109364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.109530 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.110071 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.110079 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.111490 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.113178 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.113184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.113186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.113407 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.113575 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.113575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.113575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.113674 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.113984 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.113988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.114113 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.114202 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.114328 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 12:58:41.115113 17872 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:41.115235 17872 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.115423 17872 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:41.115819 17413 ts_manager.cc:194] Registered new tserver with Master: 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:41.116259 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:58363
I20260928 12:58:41.120733 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.11.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.125.keytab.
Entry for principal oryx/127.11.11.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.125.keytab.
I20260928 12:58:41.147284 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:37313
--webserver_interface=127.11.11.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.125.keytab
--principal=oryx/127.11.11.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:42529,127.11.11.125:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:41.233134 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:41.233369 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:41.233394 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:41.234651 17877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:41.234700 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:41.234715 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:41.234728 17877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:41.234740 17877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:41.236445 17877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:42529,127.11.11.125:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.125.keytab
--principal=oryx/127.11.11.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.11.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
W20260928 12:58:41.236572 17877 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.11.126:42529,127.11.11.125:37313'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 12:58:41.236697 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:41.236882 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.241914 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.241902 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.241914 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.242414 17877 server_base.cc:1034] running on GCE node
I20260928 12:58:41.242671 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.242930 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.244110 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790600321244087 us; error 35 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.246819 17877 init.cc:377] Logged in from keytab as oryx/127.11.11.125@KRBTEST.COM (short username oryx)
I20260928 12:58:41.247298 17877 webserver.cc:460] Webserver started at http://127.11.11.125:35103/ using document root <none> and password file <none>
I20260928 12:58:41.247485 17877 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.247527 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.247628 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:41.248749 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/instance:
uuid: "5f0c147699e8405e916b1a6df697525e"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "b81107dbe36eb6ffd3928e8d0e1603c8"
server_key_iv: "ea7dd31b994cca613305c779414fedbb"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.249053 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal/instance:
uuid: "5f0c147699e8405e916b1a6df697525e"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "b81107dbe36eb6ffd3928e8d0e1603c8"
server_key_iv: "ea7dd31b994cca613305c779414fedbb"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.250383 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:41.251127 17892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.251304 17877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.251411 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "5f0c147699e8405e916b1a6df697525e"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "b81107dbe36eb6ffd3928e8d0e1603c8"
server_key_iv: "ea7dd31b994cca613305c779414fedbb"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.251489 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.267454 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.270025 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:41.270210 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.274649 17877 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.125:37313
I20260928 12:58:41.274688 17944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.125:37313 every 8 connection(s)
I20260928 12:58:41.275071 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
I20260928 12:58:41.276216 17945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:41.277976 17945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:58:41.282481 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 17877
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.125@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:41.287547 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.278733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.11.11.126:42529)
0928 12:58:41.278783 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.278785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.279003 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:58:41.279471 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.279472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.279485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.279552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.280252 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.280760 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.281748 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.281930 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.281981 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.284545 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.284547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.284547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.284555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.286052 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.286493 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.286494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.286495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.286558 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.286820 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.286821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.286821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.286985 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.287063 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.287363 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.287449 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:41.287745 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.278697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35352)
0928 12:58:41.278873 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.278916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.279382 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.279615 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.279619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.279903 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.280173 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.280178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.280839 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.280839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.281674 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.281676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.282000 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.282000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.282107 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.282751 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.282763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.284450 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.286157 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.286165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.286167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.286412 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.286623 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.286624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.286625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.286745 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.287053 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.287060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.287219 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.287327 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.287495 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
I20260928 12:58:41.288688 17945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:58:41.291635 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.283025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.11.11.125:37313)
0928 12:58:41.283190 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.283196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.283223 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:58:41.283270 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.283273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.283294 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.283381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.284012 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.284936 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.285714 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.286034 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.286115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.288573 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.288578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.288578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.288585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.290421 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.290807 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.290810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.290811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.290875 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.291046 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.291047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.291047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.291220 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.291360 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.291448 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.291510 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.125@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
I20260928 12:58:41.296149 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.288997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53162)
0928 12:58:41.289044 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.289054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.289088 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.289306 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.289308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.289486 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.289545 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.289547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.290315 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.290316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.291118 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.291120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.291572 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.291572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.291648 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.292124 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.292130 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.293367 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.295218 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.295219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.295219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.295385 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.295563 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.295564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.295564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.295612 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.295790 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.295791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.295826 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.295877 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.296012 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":57}
I20260928 12:58:41.296267 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.289055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.11.11.125:37313)
0928 12:58:41.289203 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.289204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.289209 (+ 5us) server_negotiation.cc:407] Connection header received
0928 12:58:41.289223 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.289223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.289231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.289281 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.289625 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.290240 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.291197 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.291497 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.291613 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.293444 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.293445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.293445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.293451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.295138 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.295459 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.295460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.295460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.295510 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.295650 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.295650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.295650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.295744 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.295789 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.295891 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.295970 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 12:58:41.297638 17945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Bootstrap starting.
I20260928 12:58:41.298302 17945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:41.298586 17945 log.cc:824] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:41.299324 17945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: No bootstrap required, opened a new log
I20260928 12:58:41.300845 17945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } }
I20260928 12:58:41.300977 17945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:41.301002 17945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0c147699e8405e916b1a6df697525e, State: Initialized, Role: FOLLOWER
I20260928 12:58:41.301141 17945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } }
I20260928 12:58:41.301446 17945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:41.301407 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } } }
I20260928 12:58:41.301517 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:58:41.303156 17963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:41.303237 17963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:41.304986 17877 master_runner.cc:428] Detected that this master 5f0c147699e8405e916b1a6df697525e is joining an existing cluster
I20260928 12:58:41.305045 17877 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.11.125:37313
I20260928 12:58:41.313087 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.306097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.11.11.126:42529)
0928 12:58:41.306149 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.306152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.306359 (+ 207us) server_negotiation.cc:407] Connection header received
0928 12:58:41.306436 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.306437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.306449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.306507 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.307019 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.307758 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.308646 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.308883 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.308960 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.310108 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.310110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.310110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.310118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.311768 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.312269 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.312270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.312271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.312335 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.312531 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.312532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.312532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.312653 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.312726 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.312897 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.312985 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:58:41.313120 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.306035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35358)
0928 12:58:41.306288 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.306298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.306372 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.306511 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.306512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.306833 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.306908 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.306910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.307844 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.307845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.308553 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.308555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.308975 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.308976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.309051 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.309529 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.309534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.310008 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.311868 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.311869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.311870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.312187 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.312396 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.312397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.312397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.312457 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.312727 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.312728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.312775 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.312871 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.313016 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":43,"thread_start_us":213,"threads_started":1}
I20260928 12:58:41.313977 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.306139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.11.11.125:37313)
0928 12:58:41.306189 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.306191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.306368 (+ 177us) server_negotiation.cc:407] Connection header received
0928 12:58:41.306460 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.306461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.306470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.306517 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.306958 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.307724 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.308905 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.309144 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.309243 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.310515 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.310516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.310516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.310523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.312348 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.313145 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.313146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.313147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.313197 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.313468 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.313469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.313469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.313583 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.313624 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.313809 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.313888 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:58:41.314293 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.306062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53178)
0928 12:58:41.306305 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.306316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.306390 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.306551 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.306552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.306813 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.306881 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.306883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.307823 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.307825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.308814 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.308816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.309514 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.309515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.309567 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.309992 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.309996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.310426 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.312436 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.312436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.312437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.313061 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.313352 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.313353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.313353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.313406 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.313664 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.313665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.313698 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.314094 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.314187 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":67,"thread_start_us":103,"threads_started":1}
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.126@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
I20260928 12:58:41.323858 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.315820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.11.11.125:37313)
0928 12:58:41.315855 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.315857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.316269 (+ 412us) server_negotiation.cc:407] Connection header received
0928 12:58:41.316372 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.316373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.316381 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.316425 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.316980 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.317457 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.318597 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.318790 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.318846 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.321125 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.321127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.321127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.321135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.322553 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.322969 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.322970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.322971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.323030 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.323227 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.323228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.323228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.323347 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.323415 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.323636 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.323721 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:58:41.323846 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.315775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53180)
0928 12:58:41.316162 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.316211 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.316299 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.316440 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.316443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.316811 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.316883 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.316885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.317535 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.317536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.318519 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.318522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.318863 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.318865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.318972 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.319463 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.319468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.321035 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.322668 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.322677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.322679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.322894 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.323104 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.323104 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.323105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.323168 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.323420 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.323429 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.323487 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.323596 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.323738 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":85,"threads_started":1}
I20260928 12:58:41.324492 17413 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.11.125:37313: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5f0c147699e8405e916b1a6df697525e"
member_type: NON_VOTER
last_known_addr {
host: "127.11.11.125"
port: 37313
}
attrs {
promote: true
}
}
dest_uuid: "ea609adf9d624c838c6baf7683f4e674"
cas_config_opid_index: -1
I20260928 12:58:41.324860 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } }
I20260928 12:58:41.325496 17972 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5f0c147699e8405e916b1a6df697525e (127.11.11.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } }
I20260928 12:58:41.325729 17919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:41.326525 17972 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.11.125:37313
I20260928 12:58:41.326534 17919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 12:58:41.326774 17877 master.cc:537] Master@127.11.11.125:37313 shutting down...
I20260928 12:58:41.327651 17972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:41.327697 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } }
I20260928 12:58:41.327795 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
W20260928 12:58:41.328590 17397 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 12:58:41.329013 17397 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 -> Peer 5f0c147699e8405e916b1a6df697525e (127.11.11.125:37313): Couldn't send request to peer 5f0c147699e8405e916b1a6df697525e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:58:41.330020 17877 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:41.330116 17877 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:41.330166 17877 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: stopping tablet replica
I20260928 12:58:41.332692 17877 master.cc:552] Master@127.11.11.125:37313 shutdown complete.
I20260928 12:58:41.332754 17877 master_runner.cc:305] Clearing existing system tablet
I20260928 12:58:41.332893 17877 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.332950 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.333609 17877 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.334095 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.334220 17877 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.334277 17877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.334306 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "5f0c147699e8405e916b1a6df697525e"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "b81107dbe36eb6ffd3928e8d0e1603c8"
server_key_iv: "ea7dd31b994cca613305c779414fedbb"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.334926 17877 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:41.336467 17877 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:58:41.336552 17877 log.cc:1196] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 12:58:41.336786 17877 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Deleting consensus metadata
I20260928 12:58:41.336921 17877 master_runner.cc:315] Copying system tablet from 127.11.11.126:42529
I20260928 12:58:41.337486 17877 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.126:42529
I20260928 12:58:41.343793 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.337877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.11.11.126:42529)
0928 12:58:41.337937 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.337939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.338070 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:58:41.338154 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.338155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.338166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.338232 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.338575 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.339095 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.339841 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.340083 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.340170 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.341130 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.341132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.341132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.341140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.342560 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.342977 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.342978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.342978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.343038 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.343238 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.343239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.343239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.343361 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.343434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.343607 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.343690 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:58:41.344009 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.337818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35370)
0928 12:58:41.337997 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.338007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.338087 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.338259 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.338260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.338441 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.338492 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.338494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.339187 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.339187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.339745 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.339747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.340177 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.340178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.340249 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.340719 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.340724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.341038 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.342686 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.342687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.342688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.342913 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.343109 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.343110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.343110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.343175 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.343425 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.343426 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.343474 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.343574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.343695 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260928 12:58:41.344444 17433 tablet_copy_service.cc:140] P ea609adf9d624c838c6baf7683f4e674: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5f0c147699e8405e916b1a6df697525e ({username='oryx', principal='oryx/127.11.11.125@KRBTEST.COM'} at 127.0.0.1:35370)
I20260928 12:58:41.344563 17433 tablet_copy_service.cc:161] P ea609adf9d624c838c6baf7683f4e674: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5f0c147699e8405e916b1a6df697525e at {username='oryx', principal='oryx/127.11.11.125@KRBTEST.COM'} at 127.0.0.1:35370: session id = 5f0c147699e8405e916b1a6df697525e-00000000000000000000000000000000
I20260928 12:58:41.345435 17433 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:58:41.346122 17877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:58:41.347404 17877 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: tablet copy: Starting download of 0 data blocks...
I20260928 12:58:41.347595 17877 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: tablet copy: Starting download of 1 WAL segments...
I20260928 12:58:41.348783 17877 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:58:41.349718 17433 tablet_copy_service.cc:342] P ea609adf9d624c838c6baf7683f4e674: Request end of tablet copy session 5f0c147699e8405e916b1a6df697525e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.11.125@KRBTEST.COM'} at 127.0.0.1:35370
I20260928 12:58:41.349785 17433 tablet_copy_service.cc:433] P ea609adf9d624c838c6baf7683f4e674: ending tablet copy session 5f0c147699e8405e916b1a6df697525e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5f0c147699e8405e916b1a6df697525e
I20260928 12:58:41.352178 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.353291 17986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.353322 17985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.357219 17988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.357239 17877 server_base.cc:1034] running on GCE node
I20260928 12:58:41.357414 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.357592 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.358716 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790600321358710 us; error 26 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.360183 17877 init.cc:377] Logged in from keytab as oryx/127.11.11.125@KRBTEST.COM (short username oryx)
I20260928 12:58:41.360484 17877 webserver.cc:460] Webserver started at http://127.11.11.125:34595/ using document root <none> and password file <none>
I20260928 12:58:41.360611 17877 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.360647 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.361264 17877 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.361860 17995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.361999 17877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.362062 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "5f0c147699e8405e916b1a6df697525e"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "b81107dbe36eb6ffd3928e8d0e1603c8"
server_key_iv: "ea7dd31b994cca613305c779414fedbb"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.362125 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.366283 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.367837 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.371155 17877 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.125:37313
I20260928 12:58:41.371527 18047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.125:37313 every 8 connection(s)
I20260928 12:58:41.371570 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
I20260928 12:58:41.372391 18048 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:58:41.372777 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Bootstrap starting.
I20260928 12:58:41.375204 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:58:41.375428 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: Bootstrap complete.
I20260928 12:58:41.375622 18048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } }
I20260928 12:58:41.375777 18048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5f0c147699e8405e916b1a6df697525e, State: Initialized, Role: LEARNER
I20260928 12:58:41.375823 18048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } }
I20260928 12:58:41.376012 18048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:41.376009 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } }
I20260928 12:58:41.376196 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: LEARNER
I20260928 12:58:41.377525 18062 catalog_manager.cc:1259] Loaded cluster ID: dbe20a57464a422e89bde1a8a8d26844
I20260928 12:58:41.377588 18062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: loading cluster ID for follower catalog manager: success
I20260928 12:58:41.378573 18062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: acquiring CA information for follower catalog manager: success
I20260928 12:58:41.378861 18062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:58:41.400171 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.393408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.11.11.125:37313)
0928 12:58:41.393560 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.393562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.393575 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:58:41.393645 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.393645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.393656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.393738 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.394309 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.394993 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.395751 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.395951 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.396024 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.397415 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.397417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.397418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.397429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.398936 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.399311 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.399313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.399313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.399373 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.399566 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.399567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.399567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.399692 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.399801 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.399909 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.400002 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.11.11.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.124.keytab.
Entry for principal oryx/127.11.11.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.124.keytab.
I20260928 12:58:41.431483 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:46791
--webserver_interface=127.11.11.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.124.keytab
--principal=oryx/127.11.11.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:41.517554 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:41.517814 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:41.517853 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:41.519039 18066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:41.519106 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:41.519131 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:41.519155 18066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:41.519179 18066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:41.520747 18066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.124.keytab
--principal=oryx/127.11.11.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.11.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:41.521031 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:41.521258 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.526487 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.526533 18066 server_base.cc:1034] running on GCE node
W20260928 12:58:41.526507 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.526494 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.526895 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.527163 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.528327 18066 hybrid_clock.cc:648] HybridClock initialized: now 1790600321528299 us; error 38 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.535853 18066 init.cc:377] Logged in from keytab as oryx/127.11.11.124@KRBTEST.COM (short username oryx)
I20260928 12:58:41.536393 18066 webserver.cc:460] Webserver started at http://127.11.11.124:34951/ using document root <none> and password file <none>
I20260928 12:58:41.536695 18066 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.536767 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.536911 18066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:41.538048 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/instance:
uuid: "de6eaf6b5bf84a3a952305c2913d0521"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "274b288bfc516ef4783375d19704cf7e"
server_key_iv: "435d830c214247fdd98eb226fe6b2977"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.538422 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal/instance:
uuid: "de6eaf6b5bf84a3a952305c2913d0521"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "274b288bfc516ef4783375d19704cf7e"
server_key_iv: "435d830c214247fdd98eb226fe6b2977"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.539777 18066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:41.540597 18081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.540774 18066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.540869 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "de6eaf6b5bf84a3a952305c2913d0521"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "274b288bfc516ef4783375d19704cf7e"
server_key_iv: "435d830c214247fdd98eb226fe6b2977"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.540953 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.552485 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.554394 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:41.554556 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.559785 18066 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.124:46791
I20260928 12:58:41.559909 18133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.124:46791 every 8 connection(s)
I20260928 12:58:41.560328 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
I20260928 12:58:41.561398 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:41.563017 18134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:58:41.566927 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18066
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.124@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
I20260928 12:58:41.573810 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.563916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.11.11.126:42529)
0928 12:58:41.563982 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.563984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.564123 (+ 139us) server_negotiation.cc:407] Connection header received
0928 12:58:41.564576 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.564577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.564591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.564652 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.565238 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.565728 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.566869 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.567109 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.567181 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.570440 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.570442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.570443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.570451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.572098 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.572524 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.572525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.572526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.572589 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.572972 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.572973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.572973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.573106 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.573193 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.573613 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.573701 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:58:41.573966 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.563726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35386)
0928 12:58:41.563969 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.564012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.564495 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.564705 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.564708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.564967 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.565160 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.565164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.565805 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.565806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.566651 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.566653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.567275 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.567276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.567503 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.568403 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.568412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.570342 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.572194 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.572200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.572202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.572433 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.572750 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.572751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.572752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.572897 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.573197 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.573204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.573390 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.573512 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.573679 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":167,"threads_started":1}
I20260928 12:58:41.574879 18134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:58:41.576184 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.567495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.11.11.124:46791)
0928 12:58:41.567745 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.567750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.567767 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:58:41.567784 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.567786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.567800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.567889 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.568515 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.569175 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.569918 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.570275 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.570355 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.573086 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.573093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.573094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.573099 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.574882 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.575272 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.575275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.575276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.575333 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.575526 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.575528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.575528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.575679 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.575835 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.575880 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.575977 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":183,"threads_started":1}
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.124@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
I20260928 12:58:41.581931 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.575184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53196)
0928 12:58:41.575232 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.575243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.575296 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.575456 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.575457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.575586 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.575726 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.575728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.576822 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.576823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.577460 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.577462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.577897 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.577897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.578017 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.578435 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.578440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.579697 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.581079 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.581079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.581080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.581236 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.581402 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.581403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.581403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.581453 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.581674 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.581675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.581714 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.581779 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.581855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:58:41.581934 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.575265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.11.11.125:37313)
0928 12:58:41.575329 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.575331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.575340 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:41.575369 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.575369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.575383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.575453 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.575808 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.576663 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.577594 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.577807 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.577899 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.579773 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.579774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.579774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.579781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.581007 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.581299 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.581299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.581300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.581348 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.581503 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.581504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.581504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.581601 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.581655 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.581806 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.581873 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:41.582378 18134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.124@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
I20260928 12:58:41.588524 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.582666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53214)
0928 12:58:41.582710 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.582719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.582754 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.582865 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.582866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.582970 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.583002 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.583003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.583681 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.583682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.584224 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.584226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.584650 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.584651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.584689 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.585103 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.585107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.586201 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.587533 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.587534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.587534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.587708 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.587863 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.587864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.587864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.587916 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.588201 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.588202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.588254 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.588318 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.588429 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:58:41.588588 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.582703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.11.11.124:46791)
0928 12:58:41.582747 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.582749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.582757 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:41.582801 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.582802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.582811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.582848 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.583049 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.583608 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.584302 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.584586 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.584643 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.586269 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.586270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.586270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.586276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.587460 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.587768 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.587768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.587769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.587811 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.587965 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.587965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.587965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.588127 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.588183 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.588347 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.588475 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":27}
I20260928 12:58:41.590394 18134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Bootstrap starting.
I20260928 12:58:41.591615 18134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:41.592326 18134 log.cc:824] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:41.593235 18134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: No bootstrap required, opened a new log
I20260928 12:58:41.596084 18134 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } }
I20260928 12:58:41.596240 18134 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:41.596263 18134 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6eaf6b5bf84a3a952305c2913d0521, State: Initialized, Role: FOLLOWER
I20260928 12:58:41.596388 18134 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } }
I20260928 12:58:41.596756 18134 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:41.596832 18139 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } } }
I20260928 12:58:41.596975 18139 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:58:41.599151 18150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:58:41.599208 18150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:58:41.601632 18066 master_runner.cc:428] Detected that this master de6eaf6b5bf84a3a952305c2913d0521 is joining an existing cluster
I20260928 12:58:41.601694 18066 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.11.124:46791
I20260928 12:58:41.610903 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.603303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35394)
0928 12:58:41.603654 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.603674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.603797 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.604010 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.604011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.604279 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.604360 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.604362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.605286 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.605287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.606387 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.606389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.606892 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.606958 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.607456 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.607462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.607781 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.609672 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.609673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.609674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.609908 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.610146 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.610148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.610148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.610234 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.610534 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.610535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.610596 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.610671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.610777 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":271,"threads_started":1}
I20260928 12:58:41.610977 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.603216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.11.11.126:42529)
0928 12:58:41.603263 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.603265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.603768 (+ 503us) server_negotiation.cc:407] Connection header received
0928 12:58:41.603876 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.603877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.603889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.603919 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.604445 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.605208 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606474 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.606789 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606862 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.607869 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.607871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.607871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.607879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.609568 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.610003 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.610004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.610005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.610073 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.610309 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.610310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.610310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.610444 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.610508 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.610806 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.610880 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:41.614462 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.604108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.11.11.124:46791)
0928 12:58:41.604147 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.604149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.608471 (+ 4322us) server_negotiation.cc:407] Connection header received
0928 12:58:41.608589 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.608589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.608599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.608662 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.609190 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.609637 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.610486 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.610778 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.610844 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.611796 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.611797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.611797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.611803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.613279 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.613646 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.613647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.613648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.613733 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.613922 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.613922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.613923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.614069 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.614111 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.614297 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.614369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:58:41.614769 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.604243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53228)
0928 12:58:41.608403 (+ 4160us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.608413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.608482 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.608678 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.608679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.609049 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.609104 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.609106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.609710 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.609711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.610411 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.610413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.610889 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.610890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.610946 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.611326 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.611330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.611698 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.613369 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.613370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.613371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.613531 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.613809 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.613810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.613810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.613860 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.614166 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.614166 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.614201 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.614578 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.614659 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4142,"thread_start_us":88,"threads_started":1}
I20260928 12:58:41.628336 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17475
I20260928 12:58:41.628599 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.603169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.11.11.125:37313)
0928 12:58:41.603267 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.603269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.604016 (+ 747us) server_negotiation.cc:407] Connection header received
0928 12:58:41.604160 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.604161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.604172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.604218 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.604951 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.605728 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606673 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.606909 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606966 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.625452 (+ 18486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.625455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.625455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.625464 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.627348 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.627684 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.627685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.627686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.627745 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.627966 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.627967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.627967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.628129 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.628173 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.628382 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.628470 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 12:58:41.629449 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.603737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53202)
0928 12:58:41.603946 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.603957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.604072 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.604313 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.604314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.604610 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.604856 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.604859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.605823 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.605824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.606598 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.606599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.606990 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.606991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.607062 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.624845 (+ 17783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.624852 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.625347 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.627426 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.627427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.627427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.627612 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.627808 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.627809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.627809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.627871 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.628222 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.628223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.628265 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.629155 (+ 890us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.629285 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600320, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.126@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
I20260928 12:58:41.633702 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:36417
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:42529
--webserver_port=41395
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:58:41.636965 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.625708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.11.11.124:46791)
0928 12:58:41.625763 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.625765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.625773 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:41.625813 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.625814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.625823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.625899 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.626203 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.626912 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.627928 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.628240 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.628301 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.634041 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.634043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.634043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.634050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.635681 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.636138 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.636139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.636140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.636198 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.636407 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.636408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.636408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.636543 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.636613 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.636788 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.636865 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:58:41.637285 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.625453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53230)
0928 12:58:41.625519 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.625531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.625571 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.625909 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.625911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.626072 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.626120 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.626122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.627003 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.627004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.627851 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.627853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.628353 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.628354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.628424 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.632383 (+ 3959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.632389 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.633897 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.635772 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.635774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.635774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.636020 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.636269 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.636270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.636270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.636345 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.636614 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.636615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.636669 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.636744 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.636870 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 12:58:41.637945 17413 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.11.124:46791: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521"
member_type: NON_VOTER
last_known_addr {
host: "127.11.11.124"
port: 46791
}
attrs {
promote: true
}
}
dest_uuid: "ea609adf9d624c838c6baf7683f4e674"
cas_config_opid_index: 5
I20260928 12:58:41.638201 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } }
I20260928 12:58:41.638715 17972 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.639130 18108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:41.639652 17972 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.11.124:46791
I20260928 12:58:41.639729 17972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.639789 17972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:41.640290 18108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:58:41.640311 18066 master.cc:537] Master@127.11.11.124:46791 shutting down...
I20260928 12:58:41.640602 17973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:58:41.641693 17397 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 -> Peer de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124:46791): Couldn't send request to peer de6eaf6b5bf84a3a952305c2913d0521. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:58:41.644337 18066 raft_consensus.cc:2229] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:41.644450 18066 raft_consensus.cc:2258] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:41.644515 18066 tablet_replica.cc:330] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: stopping tablet replica
I20260928 12:58:41.647154 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.638701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53210)
0928 12:58:41.641770 (+ 3069us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.641782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.641854 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.642101 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.642102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.642104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:41.642152 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.642154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.643854 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.643855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.645945 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.645949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.646878 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.646879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.646948 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.646975 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.647030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.647079 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3051}
I20260928 12:58:41.647195 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.638804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.11.11.125:37313)
0928 12:58:41.638848 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.638850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.641826 (+ 2976us) server_negotiation.cc:407] Connection header received
0928 12:58:41.641931 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.641933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.641957 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.642000 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:41.642681 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.643561 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.646075 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.646688 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.646737 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.646817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.647078 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.647124 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:41.647408 18022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 LEARNER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:58:41.647934 17973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 12:58:41.648654 18022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.650038 17973 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: attempting to promote NON_VOTER 5f0c147699e8405e916b1a6df697525e to VOTER
I20260928 12:58:41.650089 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea609adf9d624c838c6baf7683f4e674. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.650171 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: LEARNER
I20260928 12:58:41.650206 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: NON_VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: true } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.650245 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: LEARNER
I20260928 12:58:41.650257 17973 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } }
I20260928 12:58:41.650743 18022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 LEARNER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 12:58:41.650750 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.650599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53244)
0928 12:58:41.650657 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.650719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:58:41.651029 17974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 12:58:41.651306 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.651215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53248)
0928 12:58:41.651257 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.651285 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:46791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:58:41.651440 17397 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 -> Peer de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124:46791): Couldn't send request to peer de6eaf6b5bf84a3a952305c2913d0521. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.124:46791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:58:41.651764 17974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5f0c147699e8405e916b1a6df697525e (127.11.11.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.651906 18022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5f0c147699e8405e916b1a6df697525e (127.11.11.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.652791 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.652868 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:41.652915 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.652956 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:41.653124 18161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.653231 18161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:58:41.657575 18066 master.cc:552] Master@127.11.11.124:46791 shutdown complete.
I20260928 12:58:41.657632 18066 master_runner.cc:305] Clearing existing system tablet
I20260928 12:58:41.657776 18066 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.657827 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.658574 18066 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.659121 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.659266 18066 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.659329 18066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.659354 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "de6eaf6b5bf84a3a952305c2913d0521"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "274b288bfc516ef4783375d19704cf7e"
server_key_iv: "435d830c214247fdd98eb226fe6b2977"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.660167 18066 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:41.661813 18066 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:58:41.661886 18066 log.cc:1196] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 12:58:41.662091 18066 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Deleting consensus metadata
I20260928 12:58:41.662213 18066 master_runner.cc:315] Copying system tablet from 127.11.11.126:42529
I20260928 12:58:41.662823 18066 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.126:42529
I20260928 12:58:41.670169 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.663200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.11.11.126:42529)
0928 12:58:41.663251 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.663253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.663391 (+ 138us) server_negotiation.cc:407] Connection header received
0928 12:58:41.663460 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.663462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.663475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.663541 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.663903 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.664699 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.665751 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.666043 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.666120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.667197 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.667199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.667199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.667207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.668923 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.669315 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.669316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.669316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.669379 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.669600 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.669601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.669601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.669741 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.669811 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.669975 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.670078 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:58:41.670179 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.663207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35398)
0928 12:58:41.663315 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.663328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.663404 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.663555 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.663555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.663766 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.663829 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.663831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.664787 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.664788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.665651 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.665653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.666128 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.666129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.666198 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.666703 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.666709 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.667105 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.669007 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.669008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.669008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.669235 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.669448 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.669449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.669449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.669531 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.669810 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.669811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.669857 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.669930 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.670040 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":97,"threads_started":1}
I20260928 12:58:41.670814 17433 tablet_copy_service.cc:140] P ea609adf9d624c838c6baf7683f4e674: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer de6eaf6b5bf84a3a952305c2913d0521 ({username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:35398)
I20260928 12:58:41.670881 17433 tablet_copy_service.cc:161] P ea609adf9d624c838c6baf7683f4e674: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer de6eaf6b5bf84a3a952305c2913d0521 at {username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:35398: session id = de6eaf6b5bf84a3a952305c2913d0521-00000000000000000000000000000000
I20260928 12:58:41.671470 17433 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:58:41.671950 18066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:58:41.673300 18066 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: tablet copy: Starting download of 0 data blocks...
I20260928 12:58:41.673457 18066 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: tablet copy: Starting download of 1 WAL segments...
I20260928 12:58:41.674326 18066 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:58:41.675264 17433 tablet_copy_service.cc:342] P ea609adf9d624c838c6baf7683f4e674: Request end of tablet copy session de6eaf6b5bf84a3a952305c2913d0521-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:35398
I20260928 12:58:41.675324 17433 tablet_copy_service.cc:433] P ea609adf9d624c838c6baf7683f4e674: ending tablet copy session de6eaf6b5bf84a3a952305c2913d0521-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer de6eaf6b5bf84a3a952305c2913d0521
I20260928 12:58:41.677822 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.678891 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.679044 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.679330 18066 server_base.cc:1034] running on GCE node
W20260928 12:58:41.679513 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.679669 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.679818 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.680939 18066 hybrid_clock.cc:648] HybridClock initialized: now 1790600321680929 us; error 23 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.682556 18066 init.cc:377] Logged in from keytab as oryx/127.11.11.124@KRBTEST.COM (short username oryx)
I20260928 12:58:41.682829 18066 webserver.cc:460] Webserver started at http://127.11.11.124:43417/ using document root <none> and password file <none>
I20260928 12:58:41.682940 18066 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.682967 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.683702 18066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:41.684321 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.684479 18066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.684527 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "de6eaf6b5bf84a3a952305c2913d0521"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "274b288bfc516ef4783375d19704cf7e"
server_key_iv: "435d830c214247fdd98eb226fe6b2977"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.684577 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.719640 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.721765 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.724952 18066 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.124:46791
I20260928 12:58:41.724992 18232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.124:46791 every 8 connection(s)
I20260928 12:58:41.725481 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
I20260928 12:58:41.726505 18233 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:58:41.727010 18233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Bootstrap starting.
I20260928 12:58:41.730317 18233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:58:41.730540 18233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Bootstrap complete.
I20260928 12:58:41.730737 18233 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } }
I20260928 12:58:41.730831 18233 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de6eaf6b5bf84a3a952305c2913d0521, State: Initialized, Role: LEARNER
I20260928 12:58:41.730866 18233 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } }
I20260928 12:58:41.731065 18233 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:41.731055 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:41.731292 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:58:41.732786 18250 catalog_manager.cc:1259] Loaded cluster ID: dbe20a57464a422e89bde1a8a8d26844
I20260928 12:58:41.733393 18250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: loading cluster ID for follower catalog manager: success
W20260928 12:58:41.733451 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:41.733678 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:41.733721 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 12:58:41.734555 18250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: acquiring CA information for follower catalog manager: success
I20260928 12:58:41.734884 18250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 12:58:41.735131 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:41.735184 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:41.735239 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:41.736848 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:36417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=41395
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:41.737111 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:41.737350 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.742370 18255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.742362 18253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.742362 18252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.742826 18159 server_base.cc:1034] running on GCE node
I20260928 12:58:41.743048 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.743290 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.744462 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790600321744443 us; error 30 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.747239 18159 init.cc:377] Logged in from keytab as oryx/127.11.11.65@KRBTEST.COM (short username oryx)
I20260928 12:58:41.747802 18159 webserver.cc:460] Webserver started at http://127.11.11.65:41395/ using document root <none> and password file <none>
I20260928 12:58:41.748107 18159 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.748189 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.750020 18159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:41.750691 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.750872 18159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:41.750958 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "e42cc89910ff2ddb800f0aa7284530b9"
server_key_iv: "b7b4f01c1ea949755cffed099e677c52"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.751320 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.767669 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.769930 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:41.770104 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.770349 18159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:41.770660 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:41.770711 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.770766 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:41.770793 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.776990 18159 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:36417
I20260928 12:58:41.777052 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:36417 every 8 connection(s)
I20260928 12:58:41.777415 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:41.779412 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18159
I20260928 12:58:41.779575 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17609
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
I20260928 12:58:41.790182 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:39295
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:42529
--webserver_port=45211
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:58:41.791900 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:35053 (local address 127.11.11.125:37313)
0928 12:58:41.778513 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.778515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.778525 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:41.780901 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.780902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.780914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.780980 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.781370 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.782060 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783388 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.783702 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783753 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.788026 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.788028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.788029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.788073 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.789959 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790663 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.790664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.790665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.790729 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.791076 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.791077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.791077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.791230 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.791272 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.791692 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.791782 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:41.791917 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:50631 (local address 127.11.11.126:42529)
0928 12:58:41.778277 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.778280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.778699 (+ 419us) server_negotiation.cc:407] Connection header received
0928 12:58:41.779953 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.779954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.779969 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.779999 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.781035 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.781847 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783383 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.783702 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783759 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.787669 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.787670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.787671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.787679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.789997 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790800 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.790801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.790802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.790861 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.791068 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.791069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.791069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.791207 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.791272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.791756 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.791834 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:58:41.792227 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.11.11.65:35747)
0928 12:58:41.781088 (+ 2584us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.781099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.781139 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.781357 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.781358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.781542 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.781622 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.781623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.782930 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.782931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.783921 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.783924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.784489 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.784490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.784534 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.785048 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.785052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.787213 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.790468 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.790470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.790863 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.791089 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.791090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.791091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.791158 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.791472 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.791472 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.791538 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.791651 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.791888 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2556,"thread_start_us":69,"threads_started":1}
I20260928 12:58:41.792337 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.11.11.65:35053)
0928 12:58:41.778397 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.778443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.779733 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.781033 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.781034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.781209 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.781268 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.781270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.782149 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.782150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.783284 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.783286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.783835 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.783983 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.784855 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.784867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.787915 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.790095 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.790098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.790547 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.790849 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.790850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.790996 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.791320 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.791320 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.791517 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.792055 (+ 538us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.792158 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260928 12:58:41.792609 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:50631)
0928 12:58:41.778620 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.778630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.779868 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.780184 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.780189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.780649 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.780946 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.780952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.781967 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.781968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.783284 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.783286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.783822 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.783823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.783983 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.784895 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.784899 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.787581 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.790083 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790095 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.790097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.790717 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.790927 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.790928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.790996 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.791299 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.791306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.791517 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.791649 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.792459 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":689,"thread_start_us":107,"threads_started":1}
I20260928 12:58:41.792965 18378 heartbeater.cc:347] Connected to a master server at 127.11.11.125:37313
I20260928 12:58:41.793069 18378 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.793083 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.778566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:35747 (local address 127.11.11.124:46791)
0928 12:58:41.781227 (+ 2661us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.781229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.781237 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:41.781259 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.781260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.781273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.781693 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.781734 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.782825 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.784020 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.784412 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.784485 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.787330 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.787332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.787332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.787340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.790361 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.790948 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.790949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.790950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.791018 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.791227 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.791228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.791229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.791384 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.791445 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.792828 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.792939 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2639,"thread_start_us":45,"threads_started":1}
I20260928 12:58:41.793298 18378 heartbeater.cc:511] Master 127.11.11.125:37313 requested a full tablet report, sending...
I20260928 12:58:41.793525 18377 heartbeater.cc:347] Connected to a master server at 127.11.11.124:46791
I20260928 12:58:41.793522 18376 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:41.793578 18377 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.793643 18376 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.793692 18377 heartbeater.cc:511] Master 127.11.11.124:46791 requested a full tablet report, sending...
I20260928 12:58:41.793728 18376 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:41.794052 18012 ts_manager.cc:194] Registered new tserver with Master: bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:41.794123 18197 ts_manager.cc:194] Registered new tserver with Master: bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:41.794417 17413 ts_manager.cc:194] Re-registered known tserver with Master: bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:41.794840 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:50631
W20260928 12:58:41.876860 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:41.877097 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:41.877151 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:41.878389 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:41.878454 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:41.878499 18384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:41.880185 18384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:39295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=45211
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:41.880462 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:41.880712 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:41.885960 18392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:41.885943 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.885986 18384 server_base.cc:1034] running on GCE node
W20260928 12:58:41.885949 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:41.886376 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:41.886662 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:41.887835 18384 hybrid_clock.cc:648] HybridClock initialized: now 1790600321887833 us; error 41 us; skew 500 ppm
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:41.890659 18384 init.cc:377] Logged in from keytab as oryx/127.11.11.66@KRBTEST.COM (short username oryx)
I20260928 12:58:41.891242 18384 webserver.cc:460] Webserver started at http://127.11.11.66:45211/ using document root <none> and password file <none>
I20260928 12:58:41.891487 18384 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:41.891568 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:41.893023 18384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:41.893615 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.893777 18384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.893882 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "253de7da96bc49d78de9c9e49bb55a2c"
format_stamp: "Formatted at 2026-09-28 12:58:40 on dist-test-slave-8sjz"
server_key: "85cc5f5e4a753f3fee583f56dc1cbee8"
server_key_iv: "b4f9636506ab6786f4140476ba646b29"
server_key_version: "encryptionkey@0"
I20260928 12:58:41.894253 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:41.917793 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:41.919839 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:41.920014 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:41.920295 18384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:41.920624 18384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:41.920678 18384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.920719 18384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:41.920766 18384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:41.927136 18384 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:39295
I20260928 12:58:41.927205 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:39295 every 8 connection(s)
I20260928 12:58:41.927589 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
Sep 28 12:58:41 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600321, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:41.937381 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18384
I20260928 12:58:41.937553 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17743
I20260928 12:58:41.939636 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.929149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:50453 (local address 127.11.11.125:37313)
0928 12:58:41.929210 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.929211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.929219 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:41.929992 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.929994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.930008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.930058 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.930846 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931649 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933311 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933490 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.934181 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.935853 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.935854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.935855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.935862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.937658 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.938210 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.938211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.938212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.938274 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.938580 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.938581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.938581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.938762 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.938808 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.939399 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.939490 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:58:41.940209 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.928915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.11.11.66:50453)
0928 12:58:41.929071 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.929083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.929829 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.930122 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.930123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.930478 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.930728 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.930735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931727 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.931727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.932812 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.932814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933579 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.933663 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.934184 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.934190 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.935750 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.937758 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.937767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.937769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.938108 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.938358 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.938358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.938359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.938500 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.939039 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.939046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.939237 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.939734 (+ 497us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.939946 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":42,"thread_start_us":92,"threads_started":1}
I20260928 12:58:41.940224 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.928431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:54521 (local address 127.11.11.124:46791)
0928 12:58:41.928487 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.928489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.928779 (+ 290us) server_negotiation.cc:407] Connection header received
0928 12:58:41.929929 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.929931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.929950 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.930011 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.930848 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931663 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.932757 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933063 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933133 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.936507 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.936513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.936513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.936521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.938799 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939270 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.939271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.939271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.939331 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939543 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.939544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.939544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.939699 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.939743 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.939956 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.940089 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:58:41.940627 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.928439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.11.11.66:54521)
0928 12:58:41.928632 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.928687 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.929801 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.930088 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.930093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.930471 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.930728 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.930735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931749 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.931750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.932655 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.932658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933148 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.933305 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.934034 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.934038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.936413 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.938896 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.938897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.938897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.939184 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.939403 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939403 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.939404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.939465 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.939790 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.939791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.939841 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.940343 (+ 502us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.940464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":61,"thread_start_us":103,"threads_started":1}
I20260928 12:58:41.940853 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.928707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:60561 (local address 127.11.11.126:42529)
0928 12:58:41.928761 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:41.928764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:41.928776 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:41.930032 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:41.930034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:41.930047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:41.930100 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:41.930838 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931410 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.932898 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933196 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933261 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.936781 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:41.936782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:41.936783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:41.936791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:41.938870 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939346 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.939347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.939348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.939411 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939616 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:41.939617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:41.939617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:41.939749 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:41.939792 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:41.940668 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 12:58:41.940763 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:41.941141 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:41.928439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:60561)
0928 12:58:41.928661 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:41.928687 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:41.929801 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:41.930104 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:41.930105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:41.930443 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:41.930727 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.930733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.931540 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.931541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.932808 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:41.932811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:41.933275 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:41.933276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:41.933326 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:41.934020 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:41.934031 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:41.936632 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:41.938958 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.938959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.938959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.939251 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.939486 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:41.939487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:41.939487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:41.939540 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:41.940175 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:41.940176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:41.940217 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:58:41.940630 (+ 413us) client_negotiation.cc:240] Negotiation successful
0928 12:58:41.941071 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":111,"thread_start_us":98,"threads_started":1}
I20260928 12:58:41.941648 18513 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:41.941757 18513 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.941936 18513 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:41.942338 17413 ts_manager.cc:194] Re-registered known tserver with Master: 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66:39295)
I20260928 12:58:41.942715 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:60561
I20260928 12:58:41.944149 18514 heartbeater.cc:347] Connected to a master server at 127.11.11.124:46791
I20260928 12:58:41.944190 18515 heartbeater.cc:347] Connected to a master server at 127.11.11.125:37313
I20260928 12:58:41.944205 18514 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.944263 18515 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:41.944296 18514 heartbeater.cc:511] Master 127.11.11.124:46791 requested a full tablet report, sending...
I20260928 12:58:41.944320 18515 heartbeater.cc:511] Master 127.11.11.125:37313 requested a full tablet report, sending...
I20260928 12:58:41.944624 18012 ts_manager.cc:194] Registered new tserver with Master: 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66:39295)
I20260928 12:58:41.944906 18197 ts_manager.cc:194] Registered new tserver with Master: 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66:39295)
I20260928 12:58:41.945412 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:40209
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:42529
--webserver_port=34271
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:42.030643 18520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:42.030925 18520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:42.030964 18520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:42.032323 18520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:42.032382 18520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:42.032425 18520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:42.033963 18520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:40209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=34271
--tserver_master_addrs=127.11.11.126:42529,127.11.11.125:37313,127.11.11.124:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:42.034261 18520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:42.034494 18520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:42.040158 18525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:42.040148 18526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:42.040249 18520 server_base.cc:1034] running on GCE node
W20260928 12:58:42.040148 18528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:42.040555 18520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:42.040834 18520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:42.042009 18520 hybrid_clock.cc:648] HybridClock initialized: now 1790600322041995 us; error 29 us; skew 500 ppm
Sep 28 12:58:42 dist-test-slave-8sjz krb5kdc[17368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600322, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:42.044581 18520 init.cc:377] Logged in from keytab as oryx/127.11.11.67@KRBTEST.COM (short username oryx)
I20260928 12:58:42.045091 18520 webserver.cc:460] Webserver started at http://127.11.11.67:34271/ using document root <none> and password file <none>
I20260928 12:58:42.045339 18520 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:42.045419 18520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:42.046793 18520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:58:42.047381 18535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:42.047546 18520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:42.047641 18520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "3a37118cdffd4b44af5bf4968552ec88"
format_stamp: "Formatted at 2026-09-28 12:58:41 on dist-test-slave-8sjz"
server_key: "1ba28c01eaa6214c63ff65ac94e637c0"
server_key_iv: "b517c5f6a57b5b98f18657533d4e9877"
server_key_version: "encryptionkey@0"
I20260928 12:58:42.048009 18520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:42.062001 18520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:42.064304 18520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:42.064484 18520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:42.064735 18520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:42.065045 18520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:42.065097 18520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:42.065138 18520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:42.065169 18520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:42.070834 18520 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40209
I20260928 12:58:42.070892 18648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40209 every 8 connection(s)
I20260928 12:58:42.071278 18520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:58:42 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600322, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.125@KRBTEST.COM
Sep 28 12:58:42 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600322, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.124@KRBTEST.COM
Sep 28 12:58:42 dist-test-slave-8sjz krb5kdc[17368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600322, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:42.080564 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18520
I20260928 12:58:42.083549 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:53961 (local address 127.11.11.125:37313)
0928 12:58:42.072631 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.072633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.072642 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:42.073526 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.073527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.073541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.073573 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.074354 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.075123 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076684 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.076999 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.077074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.079094 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.079096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.079096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.079104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.081206 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082358 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.082360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.082360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.082426 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082744 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.082745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.082745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.082892 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.082936 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.083358 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.083438 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:58:42.083770 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:46637)
0928 12:58:42.072301 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.072343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.073401 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.073651 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.073655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.074031 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:42.074243 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.074247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.074966 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.074967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.075801 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.075803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.076324 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.076471 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.077303 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:42.077307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:42.080120 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:42.082061 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.082062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.082302 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.082697 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.082699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.082754 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.083030 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:42.083031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:42.083193 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.083312 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.083493 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260928 12:58:42.084120 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:46637 (local address 127.11.11.126:42529)
0928 12:58:42.072191 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.072193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.072421 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:58:42.073504 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.073505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.073518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.073573 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.074347 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.074863 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.075932 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.076226 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076317 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.080230 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.080232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.080232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.080240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.081950 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082566 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.082567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.082568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.082622 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082822 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.082823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.082823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.082951 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.083008 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.083902 (+ 894us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.083998 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:42.084163 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:40247 (local address 127.11.11.124:46791)
0928 12:58:42.072454 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.072456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.072465 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:42.073526 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.073527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.073541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.073573 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.074354 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.075123 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076233 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.076552 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.080256 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.080257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.080258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.080265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.082735 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.083294 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.083295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.083296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.083360 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.083582 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.083583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.083583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.083710 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.083757 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.083951 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.084069 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:58:42.084439 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.11.11.67:40247)
0928 12:58:42.072300 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.072343 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.073400 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.073651 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.073655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.074091 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:42.074243 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.074250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.075223 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.075224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.076160 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.076162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.076634 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.076635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.076709 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.077297 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:42.077307 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:42.080120 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:42.082822 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.082824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.083042 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.083433 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.083433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.083434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.083495 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.083789 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:42.083790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:42.083830 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.084246 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.084350 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":36,"thread_start_us":106,"threads_started":1}
I20260928 12:58:42.084538 18649 heartbeater.cc:347] Connected to a master server at 127.11.11.126:42529
I20260928 12:58:42.084625 18649 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:42.084810 18649 heartbeater.cc:511] Master 127.11.11.126:42529 requested a full tablet report, sending...
I20260928 12:58:42.084841 18650 heartbeater.cc:347] Connected to a master server at 127.11.11.124:46791
I20260928 12:58:42.084882 18650 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:42.084933 18650 heartbeater.cc:511] Master 127.11.11.124:46791 requested a full tablet report, sending...
I20260928 12:58:42.085196 17413 ts_manager.cc:194] Re-registered known tserver with Master: 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:42.085320 18197 ts_manager.cc:194] Registered new tserver with Master: 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:42.085695 17413 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:46637
I20260928 12:58:42.086265 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.072430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.11.11.67:53961)
0928 12:58:42.072580 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.072589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.073408 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.073651 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.073655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.074096 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:42.074243 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.074250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.075223 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.075224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.076169 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.076171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.077076 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.077077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.077149 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.077484 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:42.077488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:42.078999 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:42.081301 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.081310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.081312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.081626 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.082504 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:42.082504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:42.082505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:42.082648 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:42.082986 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:42.082992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:42.083193 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.084140 (+ 947us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.084951 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":53,"thread_start_us":69,"threads_started":1}
I20260928 12:58:42.086581 18651 heartbeater.cc:347] Connected to a master server at 127.11.11.125:37313
I20260928 12:58:42.086632 18651 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:42.086711 18651 heartbeater.cc:511] Master 127.11.11.125:37313 requested a full tablet report, sending...
I20260928 12:58:42.087006 18012 ts_manager.cc:194] Registered new tserver with Master: 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:42.092245 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.088382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53264)
0928 12:58:42.088462 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.088471 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.088504 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.088784 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.088785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.088788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.088835 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.088837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.089769 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.089769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.090980 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.090983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.091946 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.091947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.092009 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.092012 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.092096 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.092160 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 12:58:42.092531 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.088576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.11.11.124:46791)
0928 12:58:42.088616 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.088618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.088624 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:58:42.088643 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.088644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.088682 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.088740 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.088909 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.089673 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.091090 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.091744 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.091802 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.091894 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.092429 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.092467 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:42.092777 18207 raft_consensus.cc:1203] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 12:58:42.092993 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea609adf9d624c838c6baf7683f4e674. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:42.093096 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:58:42.093278 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.082113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.11.11.125:37313)
0928 12:58:42.082462 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.082464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.082471 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:42.082492 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.082493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.082506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.082567 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.083424 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.084495 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.086535 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.086930 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.087330 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.088404 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.088405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.088406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.088413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.091229 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.091640 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.091641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.091642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.091700 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.091907 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.091908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.091908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.092066 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.092824 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.092922 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.093014 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":65,"threads_started":1}
I20260928 12:58:42.093461 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.082092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.11.11.124:46791)
0928 12:58:42.082235 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.082237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.085290 (+ 3053us) server_negotiation.cc:407] Connection header received
0928 12:58:42.085309 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.085310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.085320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.085374 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.086519 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.087294 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.088358 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.088655 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.088695 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.090030 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.090031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.090031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.090038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.092090 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.092476 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.092477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.092478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.092537 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.092746 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.092747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.092748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.092885 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.092975 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.093279 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.093364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 12:58:42.093932 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.081905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.11.11.126:42529)
0928 12:58:42.084209 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.084211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.086455 (+ 2244us) server_negotiation.cc:407] Connection header received
0928 12:58:42.086549 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.086550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.086560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.086588 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.087286 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.088302 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.089278 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.089600 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.089646 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.091039 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.091040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.091041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.091047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.092752 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.093115 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.093116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.093117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.093170 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.093390 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.093390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.093391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.093511 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.093584 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.093788 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.093863 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2284,"thread_start_us":83,"threads_started":1}
I20260928 12:58:42.095731 17413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:42.097512 17413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:42.098320 17972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: NON_VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: true } } }
I20260928 12:58:42.098421 17972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:42.099582 17972 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: attempting to promote NON_VOTER de6eaf6b5bf84a3a952305c2913d0521 to VOTER
I20260928 12:58:42.099730 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:42.100185 18022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 12:58:42.100461 18206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 LEARNER]: Refusing update from remote peer ea609adf9d624c838c6baf7683f4e674: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 12:58:42.100765 17973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 12:58:42.100936 17973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 12:58:42.101576 17973 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.102375 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.102473 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:42.102669 17972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.102736 17972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:42.102650 18022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.102371 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.102921 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:42.103752 18161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.103858 18161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:58:42.104077 18206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.105118 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea609adf9d624c838c6baf7683f4e674" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:42.105216 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:58:42.107749 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.102948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39295 (local address 127.0.0.1:60064)
0928 12:58:42.103303 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.103316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.103369 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.103738 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.103740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.103742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.103798 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.103800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.105309 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.105311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.106430 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.106432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107496 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.107497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.107574 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.107579 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.107627 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.107671 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260928 12:58:42.108191 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.103010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40209 (local address 127.0.0.1:44048)
0928 12:58:42.103816 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.103826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.103876 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.105362 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.105363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.105365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.105400 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.105401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.106524 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.106526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.107212 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.107214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107915 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.107916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.107975 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.107979 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.108010 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.108110 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":87,"threads_started":1}
I20260928 12:58:42.108214 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.103293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.11.11.66:39295)
0928 12:58:42.103468 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.103473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.103486 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:58:42.103542 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.103545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.103564 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.103655 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.104160 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.105227 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.106529 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107325 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.107388 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.107465 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.107857 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.107904 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260928 12:58:42.108624 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.103503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.11.11.67:40209)
0928 12:58:42.103702 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.103707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.103911 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:58:42.103963 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.103965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.103983 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.105375 (+ 1392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.105478 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.106438 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.107303 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107788 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.108373 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.108469 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.108495 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.108544 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260928 12:58:42.108960 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e (DEFAULT_TABLE table=test-table [id=afa4b785b82e4fc2bf219285bb313f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:42.109210 18583 tablet_service.cc:1467] Processing CreateTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e (DEFAULT_TABLE table=test-table [id=afa4b785b82e4fc2bf219285bb313f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:42.109339 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09deedf6a8974dc8a09624fe95c0ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:42.109539 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09deedf6a8974dc8a09624fe95c0ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:42.109898 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.103066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:36417 (local address 127.0.0.1:48546)
0928 12:58:42.105650 (+ 2584us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.105660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.105724 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.105977 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.105978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.105980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.106016 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.106017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107303 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.107304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.108672 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.108674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.109657 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.109658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.109730 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.109735 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.109786 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.109828 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2565,"thread_start_us":93,"threads_started":1}
I20260928 12:58:42.110373 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.103185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.11.11.65:36417)
0928 12:58:42.104073 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.104078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.105740 (+ 1662us) server_negotiation.cc:407] Connection header received
0928 12:58:42.105797 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.105799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.105818 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.106076 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.106139 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.107200 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.108767 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.109470 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.109539 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.109621 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.110074 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.110121 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":81,"threads_started":1}
I20260928 12:58:42.111007 18310 tablet_service.cc:1467] Processing CreateTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e (DEFAULT_TABLE table=test-table [id=afa4b785b82e4fc2bf219285bb313f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:42.111348 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09deedf6a8974dc8a09624fe95c0ee7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:42.112643 18678 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Bootstrap starting.
I20260928 12:58:42.113317 18678 tablet_bootstrap.cc:654] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:42.113605 18678 log.cc:824] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:42.114324 18678 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: No bootstrap required, opened a new log
I20260928 12:58:42.114404 18678 ts_tablet_manager.cc:1397] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:42.114665 18681 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Bootstrap starting.
I20260928 12:58:42.114905 18679 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Bootstrap starting.
I20260928 12:58:42.115415 18679 tablet_bootstrap.cc:654] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:42.115444 18681 tablet_bootstrap.cc:654] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:42.115697 18679 log.cc:824] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:42.115744 18681 log.cc:824] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:42.116462 18679 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: No bootstrap required, opened a new log
I20260928 12:58:42.116462 18681 tablet_bootstrap.cc:492] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: No bootstrap required, opened a new log
I20260928 12:58:42.116526 18679 ts_tablet_manager.cc:1397] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:42.116526 18681 ts_tablet_manager.cc:1397] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:42.117100 18678 raft_consensus.cc:348] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.117226 18678 raft_consensus.cc:374] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:42.117259 18678 raft_consensus.cc:729] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a37118cdffd4b44af5bf4968552ec88, State: Initialized, Role: FOLLOWER
I20260928 12:58:42.117379 18678 consensus_queue.cc:260] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.117619 18649 heartbeater.cc:503] Master 127.11.11.126:42529 was elected leader, sending a full tablet report...
I20260928 12:58:42.117842 18678 ts_tablet_manager.cc:1428] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:42.118496 18681 raft_consensus.cc:348] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.118666 18681 raft_consensus.cc:374] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:42.118711 18681 raft_consensus.cc:729] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2dea8ced8b47049a1a64c64dc2d6e2, State: Initialized, Role: FOLLOWER
I20260928 12:58:42.118827 18681 consensus_queue.cc:260] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.119095 18681 ts_tablet_manager.cc:1428] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:58:42.119216 18376 heartbeater.cc:503] Master 127.11.11.126:42529 was elected leader, sending a full tablet report...
I20260928 12:58:42.119963 18679 raft_consensus.cc:348] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.120160 18679 raft_consensus.cc:374] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:42.120208 18679 raft_consensus.cc:729] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253de7da96bc49d78de9c9e49bb55a2c, State: Initialized, Role: FOLLOWER
I20260928 12:58:42.120318 18679 consensus_queue.cc:260] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.120603 18679 ts_tablet_manager.cc:1428] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:58:42.120623 18513 heartbeater.cc:503] Master 127.11.11.126:42529 was elected leader, sending a full tablet report...
W20260928 12:58:42.179118 18516 tablet.cc:2369] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:42.209185 18686 raft_consensus.cc:482] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:42.209314 18686 raft_consensus.cc:504] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.209764 18686 leader_election.cc:288] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417), 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:42.212548 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.210010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:44813 (local address 127.11.11.67:40209)
0928 12:58:42.210060 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.210062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.210072 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:42.210135 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.210135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.210145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.210218 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.210382 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.210849 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.211807 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.212192 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.212278 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.212287 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.212437 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.212475 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:58:42.212580 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.209942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40209 (local address 127.11.11.66:44813)
0928 12:58:42.210020 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.210031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.210073 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.210220 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.210221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.210222 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.210308 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.210310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.210934 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.210935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.211711 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.211713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.212275 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.212276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.212353 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.212356 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.212428 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.212468 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":49}
I20260928 12:58:42.212571 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.209977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:57671 (local address 127.11.11.65:36417)
0928 12:58:42.210021 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.210024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.210037 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:58:42.210134 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.210135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.210149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.210216 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:42.210374 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.210848 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.211806 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.212224 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.212304 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.212313 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.212468 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.212505 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:42.212597 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.209916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:36417 (local address 127.11.11.66:57671)
0928 12:58:42.209995 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:42.210008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:42.210053 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:42.210219 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:42.210219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:42.210221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:42.210305 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.210307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.210931 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.210932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.211713 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:42.211716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.212303 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.212304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:42.212386 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.212388 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:42.212447 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:58:42.212491 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":2}
I20260928 12:58:42.212934 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09deedf6a8974dc8a09624fe95c0ee7e" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a37118cdffd4b44af5bf4968552ec88" is_pre_election: true
I20260928 12:58:42.213105 18603 raft_consensus.cc:2454] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 0.
I20260928 12:58:42.213076 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09deedf6a8974dc8a09624fe95c0ee7e" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" is_pre_election: true
I20260928 12:58:42.213230 18330 raft_consensus.cc:2454] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 0.
I20260928 12:58:42.213343 18402 leader_election.cc:302] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253de7da96bc49d78de9c9e49bb55a2c, 3a37118cdffd4b44af5bf4968552ec88; no voters:
I20260928 12:58:42.213464 18686 raft_consensus.cc:2781] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:42.213516 18686 raft_consensus.cc:482] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:42.213541 18686 raft_consensus.cc:3037] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:42.214273 18686 raft_consensus.cc:504] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.214417 18686 leader_election.cc:288] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 election: Requested vote from peers bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417), 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209)
I20260928 12:58:42.214567 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09deedf6a8974dc8a09624fe95c0ee7e" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a37118cdffd4b44af5bf4968552ec88"
I20260928 12:58:42.214627 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09deedf6a8974dc8a09624fe95c0ee7e" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
I20260928 12:58:42.214656 18603 raft_consensus.cc:3037] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:42.214726 18330 raft_consensus.cc:3037] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:42.215365 18603 raft_consensus.cc:2454] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 1.
I20260928 12:58:42.215440 18330 raft_consensus.cc:2454] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 1.
I20260928 12:58:42.215572 18402 leader_election.cc:302] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253de7da96bc49d78de9c9e49bb55a2c, 3a37118cdffd4b44af5bf4968552ec88; no voters:
I20260928 12:58:42.215708 18686 raft_consensus.cc:2781] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:42.215893 18686 raft_consensus.cc:686] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 1 LEADER]: Becoming Leader. State: Replica: 253de7da96bc49d78de9c9e49bb55a2c, State: Running, Role: LEADER
I20260928 12:58:42.216009 18686 consensus_queue.cc:237] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } }
I20260928 12:58:42.216899 17413 catalog_manager.cc:5495] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66). New cstate: current_term: 1 leader_uuid: "253de7da96bc49d78de9c9e49bb55a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:42.222818 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.220620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.11.11.66:39295)
0928 12:58:42.220662 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.220664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.220673 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:42.220710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.220711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.220725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.220784 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:42.220880 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.221344 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.222042 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.222226 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.222283 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.222404 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:42.222514 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:42.222570 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.222631 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.222648 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:42.225015 18603 raft_consensus.cc:1261] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Refusing update from remote peer 253de7da96bc49d78de9c9e49bb55a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:42.225036 18330 raft_consensus.cc:1261] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Refusing update from remote peer 253de7da96bc49d78de9c9e49bb55a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:42.225359 18687 consensus_queue.cc:1035] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:42.225499 18686 consensus_queue.cc:1035] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:42.229182 18691 mvcc.cc:205] Tried to move back new op lower bound from 7334298919831261184 to 7334298919797645312. Current Snapshot: MvccSnapshot[applied={T|T < 7334298919831261184}]
I20260928 12:58:42.229295 18690 mvcc.cc:205] Tried to move back new op lower bound from 7334298919831261184 to 7334298919797645312. Current Snapshot: MvccSnapshot[applied={T|T < 7334298919831261184 or (T in {7334298919831261184})}]
I20260928 12:58:42.230396 17412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:42.230505 17412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:42.230672 18692 mvcc.cc:205] Tried to move back new op lower bound from 7334298919831261184 to 7334298919797645312. Current Snapshot: MvccSnapshot[applied={T|T < 7334298919831261184}]
I20260928 12:58:42.232396 17412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea609adf9d624c838c6baf7683f4e674: Sending DeleteTablet for 3 replicas of tablet 09deedf6a8974dc8a09624fe95c0ee7e
I20260928 12:58:42.232632 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:42 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:48546
I20260928 12:58:42.232690 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:42 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:60064
I20260928 12:58:42.232834 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:42 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:44048
I20260928 12:58:42.232879 18309 tablet_service.cc:1514] Processing DeleteTablet for tablet 09deedf6a8974dc8a09624fe95c0ee7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:42 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:48546
I20260928 12:58:42.233022 18699 tablet_replica.cc:330] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: stopping tablet replica
I20260928 12:58:42.233053 18701 tablet_replica.cc:330] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: stopping tablet replica
I20260928 12:58:42.233070 18700 tablet_replica.cc:330] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: stopping tablet replica
I20260928 12:58:42.233156 18700 raft_consensus.cc:2229] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:42.233156 18701 raft_consensus.cc:2229] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:42.233260 18700 raft_consensus.cc:2258] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:42.233264 18701 raft_consensus.cc:2258] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:42.233269 18699 raft_consensus.cc:2229] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:42.233476 18699 raft_consensus.cc:2258] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:42.233742 18701 ts_tablet_manager.cc:1905] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:42.234130 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17381
I20260928 12:58:42.234244 18699 ts_tablet_manager.cc:1905] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:42.235518 18701 ts_tablet_manager.cc:1918] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:42.235582 18701 log.cc:1196] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/09deedf6a8974dc8a09624fe95c0ee7e
I20260928 12:58:42.235858 18699 ts_tablet_manager.cc:1918] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:42.235894 18701 ts_tablet_manager.cc:1939] T 09deedf6a8974dc8a09624fe95c0ee7e P 3a37118cdffd4b44af5bf4968552ec88: Deleting consensus metadata
I20260928 12:58:42.235932 18699 log.cc:1196] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/09deedf6a8974dc8a09624fe95c0ee7e
I20260928 12:58:42.236284 18700 ts_tablet_manager.cc:1905] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:42.237497 18700 ts_tablet_manager.cc:1918] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:42.237534 18699 ts_tablet_manager.cc:1939] T 09deedf6a8974dc8a09624fe95c0ee7e P bc2dea8ced8b47049a1a64c64dc2d6e2: Deleting consensus metadata
I20260928 12:58:42.237567 18700 log.cc:1196] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/09deedf6a8974dc8a09624fe95c0ee7e
I20260928 12:58:42.237845 18700 ts_tablet_manager.cc:1939] T 09deedf6a8974dc8a09624fe95c0ee7e P 253de7da96bc49d78de9c9e49bb55a2c: Deleting consensus metadata
W20260928 12:58:42.242662 18265 connection.cc:698] server connection from 127.0.0.1:48546 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260928 12:58:42.242738 18265 connection.cc:441] server connection from 127.0.0.1:48546 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48546 (request call id 2) could send its response
W20260928 12:58:42.243515 18536 connection.cc:535] server connection from 127.0.0.1:44048 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:58:42.242489 18401 connection.cc:535] server connection from 127.0.0.1:60064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:58:42.247639 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.233846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.11.11.124:46791)
0928 12:58:42.233889 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.233891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.240237 (+ 6346us) server_negotiation.cc:407] Connection header received
0928 12:58:42.240323 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.240324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.240341 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.240399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.240994 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.241504 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.242234 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.242536 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.242607 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.244486 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.244488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.244489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.244498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.246256 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.246676 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.246677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.246678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.246735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.246922 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.246923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.246924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.247065 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.247184 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.247459 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.247544 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:42.252696 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.245512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.11.11.125:37313)
0928 12:58:42.245617 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.245619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.245629 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:42.245711 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.245712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.245727 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.245765 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.246307 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.246793 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.247844 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.248155 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.248202 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.249487 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.249489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.249489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.249496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.251106 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.251615 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.251617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.251617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.251684 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.251897 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.251898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.251898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.252157 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.252260 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.252495 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.252595 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 12:58:42.254962 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:42.245665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.11.11.124:46791)
0928 12:58:42.245708 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:42.245710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:42.247024 (+ 1314us) server_negotiation.cc:407] Connection header received
0928 12:58:42.247122 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:42.247123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:42.247133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:42.247168 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:42.248544 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.249293 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.250301 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:42.250613 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:42.250681 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:42.252242 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:42.252243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:42.252243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:42.252250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:42.253873 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.254194 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.254195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.254195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.254247 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:42.254441 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:42.254442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:42.254443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:42.254553 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:42.254662 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:42.254772 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:58:42.254848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:58:43.229766 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.229462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:37705)
0928 12:58:43.229611 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.229703 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260928 12:58:43.229943 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.229649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:45919)
0928 12:58:43.229818 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.229897 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":47,"threads_started":1}
W20260928 12:58:43.230027 18513 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
W20260928 12:58:43.230213 18649 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:43.233191 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.232975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:47637)
0928 12:58:43.233082 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.233152 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
W20260928 12:58:43.233372 18376 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:42529 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:43.733114 18713 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea609adf9d624c838c6baf7683f4e674)
I20260928 12:58:43.733201 18714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea609adf9d624c838c6baf7683f4e674)
I20260928 12:58:43.733269 18713 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:43.733340 18714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:43.733628 18714 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124:46791)
I20260928 12:58:43.733636 18713 leader_election.cc:288] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), 5f0c147699e8405e916b1a6df697525e (127.11.11.125:37313)
I20260928 12:58:43.733980 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.733761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35946)
0928 12:58:43.733874 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.733935 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 12:58:43.734026 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.733757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35934)
0928 12:58:43.733947 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.733985 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":39,"threads_started":1}
W20260928 12:58:43.734127 17999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
W20260928 12:58:43.734223 18184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:43.734709 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.734580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35948)
0928 12:58:43.734654 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.734680 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 12:58:43.734743 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.734602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35958)
0928 12:58:43.734635 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.734675 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 12:58:43.734819 17999 leader_election.cc:334] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
W20260928 12:58:43.734905 18184 leader_election.cc:334] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:43.737241 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.733872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:37313 (local address 127.0.0.1:53226)
0928 12:58:43.734001 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.734012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:43.734067 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:43.734404 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:43.734405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:43.734408 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:43.734484 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:43.734486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.735601 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.735601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:43.736432 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:43.736434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.736989 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.736990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:43.737052 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:43.737055 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:43.737124 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:58:43.737170 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260928 12:58:43.737260 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.733913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.11.11.125:37313)
0928 12:58:43.734156 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:43.734158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:43.734167 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:43.734193 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:43.734194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:43.734206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:43.734323 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:43.734575 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.735517 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.736526 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.736899 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.736986 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:43.736997 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:43.737138 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:58:43.737182 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":38,"threads_started":1}
I20260928 12:58:43.737391 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.734083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:46791 (local address 127.0.0.1:53282)
0928 12:58:43.734352 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.734363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:43.734422 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:43.734729 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:43.734730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:43.734734 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:43.734785 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:43.734893 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.735781 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.735782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:43.736586 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:43.736589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.737149 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.737150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:43.737211 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:43.737215 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:43.737285 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:58:43.737326 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":51,"threads_started":1}
I20260928 12:58:43.737390 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.734271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.11.11.124:46791)
0928 12:58:43.734414 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:43.734416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:43.734424 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:43.734511 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:43.734512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:43.734523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:43.734686 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:43.734999 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.735706 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.736684 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:43.737057 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:43.737145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:43.737156 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:43.737292 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 12:58:43.737330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
I20260928 12:58:43.738104 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0c147699e8405e916b1a6df697525e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de6eaf6b5bf84a3a952305c2913d0521" is_pre_election: true
I20260928 12:58:43.738104 18022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de6eaf6b5bf84a3a952305c2913d0521" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0c147699e8405e916b1a6df697525e" is_pre_election: true
I20260928 12:58:43.738204 18206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f0c147699e8405e916b1a6df697525e in term 1.
I20260928 12:58:43.738224 18022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6eaf6b5bf84a3a952305c2913d0521 in term 1.
I20260928 12:58:43.738363 18181 leader_election.cc:302] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0c147699e8405e916b1a6df697525e, de6eaf6b5bf84a3a952305c2913d0521; no voters: ea609adf9d624c838c6baf7683f4e674
I20260928 12:58:43.738415 17996 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0c147699e8405e916b1a6df697525e, de6eaf6b5bf84a3a952305c2913d0521; no voters: ea609adf9d624c838c6baf7683f4e674
I20260928 12:58:43.738492 18713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:58:43.738520 18714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:58:43.738531 18713 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ea609adf9d624c838c6baf7683f4e674)
I20260928 12:58:43.738556 18713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:58:43.738565 18714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ea609adf9d624c838c6baf7683f4e674)
I20260928 12:58:43.738586 18714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:58:43.739248 18714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:43.739250 18713 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:43.739380 18714 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 election: Requested vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), de6eaf6b5bf84a3a952305c2913d0521 (127.11.11.124:46791)
I20260928 12:58:43.739385 18713 leader_election.cc:288] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 election: Requested vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), 5f0c147699e8405e916b1a6df697525e (127.11.11.125:37313)
I20260928 12:58:43.739539 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.739458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35982)
0928 12:58:43.739492 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.739517 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:58:43.739636 18022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de6eaf6b5bf84a3a952305c2913d0521" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0c147699e8405e916b1a6df697525e"
I20260928 12:58:43.739748 18022 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate de6eaf6b5bf84a3a952305c2913d0521 in current term 2: Already voted for candidate 5f0c147699e8405e916b1a6df697525e in this term.
I20260928 12:58:43.739634 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f0c147699e8405e916b1a6df697525e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de6eaf6b5bf84a3a952305c2913d0521"
I20260928 12:58:43.739782 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.739507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35988)
0928 12:58:43.739576 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.739637 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 12:58:43.739799 18206 raft_consensus.cc:2379] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f0c147699e8405e916b1a6df697525e in current term 2: Already voted for candidate de6eaf6b5bf84a3a952305c2913d0521 in this term.
I20260928 12:58:43.740100 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.739982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:35994)
0928 12:58:43.740017 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.740072 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:58:43.740160 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:43.739984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36002)
0928 12:58:43.740109 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:43.740135 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 12:58:43.740188 18184 leader_election.cc:334] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
W20260928 12:58:43.740213 17999 leader_election.cc:334] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:43.740242 18184 leader_election.cc:302] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de6eaf6b5bf84a3a952305c2913d0521; no voters: 5f0c147699e8405e916b1a6df697525e, ea609adf9d624c838c6baf7683f4e674
I20260928 12:58:43.740243 17999 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f0c147699e8405e916b1a6df697525e; no voters: de6eaf6b5bf84a3a952305c2913d0521, ea609adf9d624c838c6baf7683f4e674
I20260928 12:58:43.740315 18714 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:58:43.740317 18713 raft_consensus.cc:2726] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 12:58:44.230667 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:44.230433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:41463)
0928 12:58:44.230563 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:44.230606 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 12:58:44.230816 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:44.230526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:41537)
0928 12:58:44.230703 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:44.230741 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":57,"threads_started":1}
I20260928 12:58:44.233915 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:44.233697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:33769)
0928 12:58:44.233820 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:44.233868 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260928 12:58:45.231547 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.231298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:60205)
0928 12:58:45.231439 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.231482 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260928 12:58:45.231563 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.231335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:34635)
0928 12:58:45.231469 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.231510 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 12:58:45.234637 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.234406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:52243)
0928 12:58:45.234533 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.234585 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260928 12:58:45.654402 18729 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:45.654510 18729 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:45.654724 18729 leader_election.cc:288] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), 5f0c147699e8405e916b1a6df697525e (127.11.11.125:37313)
I20260928 12:58:45.654914 18022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de6eaf6b5bf84a3a952305c2913d0521" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0c147699e8405e916b1a6df697525e" is_pre_election: true
I20260928 12:58:45.654984 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.654809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36278)
0928 12:58:45.654907 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.654946 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260928 12:58:45.655021 18022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de6eaf6b5bf84a3a952305c2913d0521 in term 2.
I20260928 12:58:45.655177 18181 leader_election.cc:302] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0c147699e8405e916b1a6df697525e, de6eaf6b5bf84a3a952305c2913d0521; no voters:
I20260928 12:58:45.655278 18729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 12:58:45.655323 18729 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:45.655359 18729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:58:45.655457 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.655359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36288)
0928 12:58:45.655401 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.655425 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:58:45.655537 18184 leader_election.cc:334] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:45.656256 18729 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:45.656394 18729 leader_election.cc:288] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 election: Requested vote from peers ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529), 5f0c147699e8405e916b1a6df697525e (127.11.11.125:37313)
I20260928 12:58:45.656570 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.656474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36294)
0928 12:58:45.656510 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.656542 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:58:45.656545 18022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de6eaf6b5bf84a3a952305c2913d0521" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5f0c147699e8405e916b1a6df697525e"
I20260928 12:58:45.656612 18022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:58:45.656903 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.656819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36308)
0928 12:58:45.656859 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.656884 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:58:45.656993 18184 leader_election.cc:334] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
I20260928 12:58:45.657377 18022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de6eaf6b5bf84a3a952305c2913d0521 in term 3.
I20260928 12:58:45.657536 18181 leader_election.cc:302] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f0c147699e8405e916b1a6df697525e, de6eaf6b5bf84a3a952305c2913d0521; no voters: ea609adf9d624c838c6baf7683f4e674
I20260928 12:58:45.657627 18729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:58:45.657747 18729 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [term 3 LEADER]: Becoming Leader. State: Replica: de6eaf6b5bf84a3a952305c2913d0521, State: Running, Role: LEADER
I20260928 12:58:45.657838 18729 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } }
I20260928 12:58:45.658207 18732 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6eaf6b5bf84a3a952305c2913d0521. Latest consensus state: current_term: 3 leader_uuid: "de6eaf6b5bf84a3a952305c2913d0521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:45.658310 18732 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:45.658495 18734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:45.658924 18734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:45.659003 18734 catalog_manager.cc:1259] Loaded cluster ID: dbe20a57464a422e89bde1a8a8d26844
I20260928 12:58:45.659037 18734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:45.659222 18734 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:45.659308 18734 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Loaded TSK: 0
I20260928 12:58:45.659456 18734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:58:45.736872 18197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:45.738530 18197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:45.739611 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.739491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36322)
0928 12:58:45.739541 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.739577 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:58:45.739634 18022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [term 3 FOLLOWER]: Refusing update from remote peer de6eaf6b5bf84a3a952305c2913d0521: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 12:58:45.739882 18729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 12:58:45.739959 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.739871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.0.0.1:36334)
0928 12:58:45.739904 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.739930 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 12:58:45.740084 18184 consensus_peers.cc:487] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 -> Peer ea609adf9d624c838c6baf7683f4e674 (127.11.11.126:42529): Couldn't send request to peer ea609adf9d624c838c6baf7683f4e674. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:58:45.740633 18736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6eaf6b5bf84a3a952305c2913d0521. Latest consensus state: current_term: 3 leader_uuid: "de6eaf6b5bf84a3a952305c2913d0521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:45.740763 18736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:58:45.741132 18735 mvcc.cc:205] Tried to move back new op lower bound from 7334298934227378176 to 7334298933895131136. Current Snapshot: MvccSnapshot[applied={T|T < 7334298934227378176}]
I20260928 12:58:45.741256 18732 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "de6eaf6b5bf84a3a952305c2913d0521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:45.741341 18732 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:45.741324 18736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "de6eaf6b5bf84a3a952305c2913d0521" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ea609adf9d624c838c6baf7683f4e674" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42529 } } peers { permanent_uuid: "5f0c147699e8405e916b1a6df697525e" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 37313 } attrs { promote: false } } peers { permanent_uuid: "de6eaf6b5bf84a3a952305c2913d0521" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 46791 } attrs { promote: false } } }
I20260928 12:58:45.741411 18736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0c147699e8405e916b1a6df697525e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:58:45.741458 18737 mvcc.cc:205] Tried to move back new op lower bound from 7334298934227378176 to 7334298933895131136. Current Snapshot: MvccSnapshot[applied={T|T < 7334298934227378176 or (T in {7334298934227378176})}]
I20260928 12:58:45.746896 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.11.11.67:40209)
0928 12:58:45.743406 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:45.743409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:45.743420 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:45.743450 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:45.743450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:45.743464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:45.743582 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:45.743698 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.744483 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.745859 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746519 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.746619 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:45.746767 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:58:45.746822 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260928 12:58:45.746896 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40209 (local address 127.0.0.1:44062)
0928 12:58:45.743318 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.743330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:45.743384 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:45.743583 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:45.743584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:45.743586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:45.743643 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.743645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.744583 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.744584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.745754 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.745757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746620 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.746687 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.746692 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:45.746755 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:58:45.746808 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 12:58:45.746948 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.11.11.66:39295)
0928 12:58:45.743429 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:45.743431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:45.743913 (+ 482us) server_negotiation.cc:407] Connection header received
0928 12:58:45.743938 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:45.743939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:45.743948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:45.744016 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:45.744195 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.744869 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.745954 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746540 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746586 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.746601 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:45.746822 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 12:58:45.746869 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 12:58:45.747015 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:36417 (local address 127.0.0.1:48550)
0928 12:58:45.743823 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.743831 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:45.743864 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:45.744232 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:45.744233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:45.744234 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:45.744268 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.744269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.745091 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.745091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.746181 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.746183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746791 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.746841 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.746844 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:45.746872 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:58:45.746905 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260928 12:58:45.747192 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39295 (local address 127.0.0.1:60078)
0928 12:58:45.743665 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.743675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:45.743743 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:45.744017 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:45.744017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:45.744019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:45.744094 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.744095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.744946 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.744947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.745818 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:45.745821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746661 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:45.746722 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.746725 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:58:45.747021 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 12:58:45.747090 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":31,"thread_start_us":79,"threads_started":1}
I20260928 12:58:45.747221 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.743501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.11.11.65:36417)
0928 12:58:45.744126 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:45.744128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:45.744136 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:45.744164 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:45.744164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:45.744175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:45.744233 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:45.744317 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.744992 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.746261 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.746697 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.747086 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.747100 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:45.747120 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 12:58:45.747157 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":56,"threads_started":1}
I20260928 12:58:45.747503 18583 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c (DEFAULT_TABLE table=test-table [id=b1adab90604446d69f1362c9763fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:45.747686 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:45.747733 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c (DEFAULT_TABLE table=test-table [id=b1adab90604446d69f1362c9763fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:45.747786 18309 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c (DEFAULT_TABLE table=test-table [id=b1adab90604446d69f1362c9763fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:45.747830 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:45.747893 18309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:45.749184 18746 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: Bootstrap starting.
I20260928 12:58:45.749404 18748 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Bootstrap starting.
I20260928 12:58:45.749573 18746 tablet_bootstrap.cc:654] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:45.749806 18748 tablet_bootstrap.cc:654] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:45.750094 18747 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Bootstrap starting.
I20260928 12:58:45.750259 18746 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: No bootstrap required, opened a new log
I20260928 12:58:45.750335 18746 ts_tablet_manager.cc:1397] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:45.750465 18747 tablet_bootstrap.cc:654] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:45.750484 18746 raft_consensus.cc:348] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.750540 18746 raft_consensus.cc:374] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:45.750553 18746 raft_consensus.cc:729] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2dea8ced8b47049a1a64c64dc2d6e2, State: Initialized, Role: FOLLOWER
I20260928 12:58:45.750588 18746 consensus_queue.cc:260] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.750694 18746 ts_tablet_manager.cc:1428] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:45.750874 18748 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: No bootstrap required, opened a new log
I20260928 12:58:45.750953 18748 ts_tablet_manager.cc:1397] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:45.751101 18748 raft_consensus.cc:348] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.751178 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.750963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:52993)
0928 12:58:45.751089 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.751125 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":84,"threads_started":1}
I20260928 12:58:45.751267 18748 raft_consensus.cc:374] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:45.751319 18748 raft_consensus.cc:729] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a37118cdffd4b44af5bf4968552ec88, State: Initialized, Role: FOLLOWER
I20260928 12:58:45.751428 18748 consensus_queue.cc:260] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.751660 18748 ts_tablet_manager.cc:1428] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:45.751672 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.751595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:41979)
0928 12:58:45.751625 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.751649 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 12:58:45.752269 18747 tablet_bootstrap.cc:492] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: No bootstrap required, opened a new log
I20260928 12:58:45.752331 18650 heartbeater.cc:503] Master 127.11.11.124:46791 was elected leader, sending a full tablet report...
I20260928 12:58:45.752351 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.751880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:37517)
0928 12:58:45.752279 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.752314 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":99,"threads_started":1}
I20260928 12:58:45.752375 18747 ts_tablet_manager.cc:1397] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:58:45.752516 18747 raft_consensus.cc:348] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.752635 18747 raft_consensus.cc:374] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:45.752698 18747 raft_consensus.cc:729] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253de7da96bc49d78de9c9e49bb55a2c, State: Initialized, Role: FOLLOWER
I20260928 12:58:45.752755 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.752561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:48299)
0928 12:58:45.752701 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.752726 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 12:58:45.752799 18747 consensus_queue.cc:260] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.752962 18747 ts_tablet_manager.cc:1428] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:45.753371 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.753129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:35945)
0928 12:58:45.753293 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.753330 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260928 12:58:45.753530 18514 heartbeater.cc:503] Master 127.11.11.124:46791 was elected leader, sending a full tablet report...
I20260928 12:58:45.753881 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.753660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:51639)
0928 12:58:45.753789 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.753828 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":91,"threads_started":1}
I20260928 12:58:45.765321 18757 raft_consensus.cc:482] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:45.765460 18757 raft_consensus.cc:504] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.765686 18757 leader_election.cc:288] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209), bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:45.765882 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc826b40c154ddbb4742ecc8b2e7e7c" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a37118cdffd4b44af5bf4968552ec88" is_pre_election: true
I20260928 12:58:45.765915 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc826b40c154ddbb4742ecc8b2e7e7c" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" is_pre_election: true
I20260928 12:58:45.766005 18603 raft_consensus.cc:2454] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 0.
I20260928 12:58:45.766001 18330 raft_consensus.cc:2454] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 0.
I20260928 12:58:45.766175 18402 leader_election.cc:302] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253de7da96bc49d78de9c9e49bb55a2c, 3a37118cdffd4b44af5bf4968552ec88; no voters:
I20260928 12:58:45.766280 18757 raft_consensus.cc:2781] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:45.766316 18757 raft_consensus.cc:482] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:45.766347 18757 raft_consensus.cc:3037] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:45.767112 18757 raft_consensus.cc:504] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.767282 18757 leader_election.cc:288] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 election: Requested vote from peers 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209), bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417)
I20260928 12:58:45.767473 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc826b40c154ddbb4742ecc8b2e7e7c" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a37118cdffd4b44af5bf4968552ec88"
I20260928 12:58:45.767500 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc826b40c154ddbb4742ecc8b2e7e7c" candidate_uuid: "253de7da96bc49d78de9c9e49bb55a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2"
I20260928 12:58:45.767556 18603 raft_consensus.cc:3037] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:45.767565 18330 raft_consensus.cc:3037] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:45.768337 18330 raft_consensus.cc:2454] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 1.
I20260928 12:58:45.768337 18603 raft_consensus.cc:2454] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 253de7da96bc49d78de9c9e49bb55a2c in term 1.
I20260928 12:58:45.768553 18400 leader_election.cc:302] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 253de7da96bc49d78de9c9e49bb55a2c, bc2dea8ced8b47049a1a64c64dc2d6e2; no voters:
I20260928 12:58:45.768663 18757 raft_consensus.cc:2781] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:45.768860 18757 raft_consensus.cc:686] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 1 LEADER]: Becoming Leader. State: Replica: 253de7da96bc49d78de9c9e49bb55a2c, State: Running, Role: LEADER
I20260928 12:58:45.768955 18757 consensus_queue.cc:237] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } }
I20260928 12:58:45.769164 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.769056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:59495)
0928 12:58:45.769097 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.769124 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 12:58:45.769572 18197 catalog_manager.cc:5495] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66). New cstate: current_term: 1 leader_uuid: "253de7da96bc49d78de9c9e49bb55a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "253de7da96bc49d78de9c9e49bb55a2c" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:45.774926 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.772807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.11.11.66:39295)
0928 12:58:45.772848 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:45.772850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:45.772856 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:58:45.772875 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:45.772876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:45.772884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:45.772929 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:45.773035 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.773531 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.774274 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:45.774460 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:45.774515 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:45.774689 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:45.774727 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:45.774790 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:45.774839 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 12:58:45.774862 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:58:45.775913 18603 raft_consensus.cc:1261] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Refusing update from remote peer 253de7da96bc49d78de9c9e49bb55a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:45.775970 18330 raft_consensus.cc:1261] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2 [term 1 FOLLOWER]: Refusing update from remote peer 253de7da96bc49d78de9c9e49bb55a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:45.776228 18760 consensus_queue.cc:1035] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a37118cdffd4b44af5bf4968552ec88" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:45.776368 18757 consensus_queue.cc:1035] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2dea8ced8b47049a1a64c64dc2d6e2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:45.776927 18377 heartbeater.cc:503] Master 127.11.11.124:46791 was elected leader, sending a full tablet report...
I20260928 12:58:45.777326 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.777206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:41545)
0928 12:58:45.777269 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.777304 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 12:58:45.778482 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.778313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:58087)
0928 12:58:45.778407 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.778448 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 12:58:45.779021 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.777206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:35379)
0928 12:58:45.778918 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.778982 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1694}
I20260928 12:58:45.779405 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.779319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.67:56723)
0928 12:58:45.779357 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.779379 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 12:58:45.780200 18379 tablet.cc:2369] T 5dc826b40c154ddbb4742ecc8b2e7e7c P bc2dea8ced8b47049a1a64c64dc2d6e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:45.780485 18759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.780097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.66:42463)
0928 12:58:45.780415 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.780449 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
I20260928 12:58:45.780931 18197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:45.781035 18197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:45.781215 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.781102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:42529 (local address 127.11.11.65:53905)
0928 12:58:45.781159 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.781186 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:58:45.782389 18197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de6eaf6b5bf84a3a952305c2913d0521: Sending DeleteTablet for 3 replicas of tablet 5dc826b40c154ddbb4742ecc8b2e7e7c
I20260928 12:58:45.782672 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:45 UTC) from {username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:60078
I20260928 12:58:45.782706 18309 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:45 UTC) from {username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:48550
I20260928 12:58:45.782886 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:45 UTC) from {username='oryx', principal='oryx/127.11.11.124@KRBTEST.COM'} at 127.0.0.1:44062
I20260928 12:58:45.782897 18774 tablet_replica.cc:330] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: stopping tablet replica
I20260928 12:58:45.782971 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18159
I20260928 12:58:45.783079 18774 raft_consensus.cc:2229] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:45.784098 18774 raft_consensus.cc:2258] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:45.784471 18774 ts_tablet_manager.cc:1905] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:45.786722 18774 ts_tablet_manager.cc:1918] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:45.786799 18774 log.cc:1196] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/5dc826b40c154ddbb4742ecc8b2e7e7c
I20260928 12:58:45.787066 18774 ts_tablet_manager.cc:1939] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 253de7da96bc49d78de9c9e49bb55a2c: Deleting consensus metadata
I20260928 12:58:45.787407 18775 tablet_replica.cc:330] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: stopping tablet replica
I20260928 12:58:45.787403 18181 catalog_manager.cc:4841] TS 253de7da96bc49d78de9c9e49bb55a2c (127.11.11.66:39295): tablet 5dc826b40c154ddbb4742ecc8b2e7e7c (table test-table [id=b1adab90604446d69f1362c9763fa3b7]) successfully deleted
I20260928 12:58:45.787489 18775 raft_consensus.cc:2229] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:45.787554 18775 raft_consensus.cc:2258] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:45.787919 18775 ts_tablet_manager.cc:1905] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:45.789953 18775 ts_tablet_manager.cc:1918] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:45.790030 18775 log.cc:1196] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/5dc826b40c154ddbb4742ecc8b2e7e7c
I20260928 12:58:45.790316 18775 ts_tablet_manager.cc:1939] T 5dc826b40c154ddbb4742ecc8b2e7e7c P 3a37118cdffd4b44af5bf4968552ec88: Deleting consensus metadata
I20260928 12:58:45.790570 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18384
I20260928 12:58:45.790596 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:45.790420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:36417 (local address 127.0.0.1:48564)
0928 12:58:45.790521 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:45.790543 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:36417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 12:58:45.790658 18183 catalog_manager.cc:4841] TS 3a37118cdffd4b44af5bf4968552ec88 (127.11.11.67:40209): tablet 5dc826b40c154ddbb4742ecc8b2e7e7c (table test-table [id=b1adab90604446d69f1362c9763fa3b7]) successfully deleted
W20260928 12:58:45.791522 18181 catalog_manager.cc:4584] TS bc2dea8ced8b47049a1a64c64dc2d6e2 (127.11.11.65:36417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dc826b40c154ddbb4742ecc8b2e7e7c: Network error: Client connection negotiation failed: client connection to 127.11.11.65:36417: connect: Connection refused (error 111)
I20260928 12:58:45.796542 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18520
I20260928 12:58:45.801605 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 17877
I20260928 12:58:45.806492 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18066
2026-09-28T12:58:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6481 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:45 dist-test-slave-8sjz krb5kdc[18778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:45 dist-test-slave-8sjz krb5kdc[18778](info): set up 2 sockets
Sep 28 12:58:45 dist-test-slave-8sjz krb5kdc[18778](info): commencing operation
krb5kdc: starting...
W20260928 12:58:46.845644 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:46 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab.
Entry for principal oryx/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab.
I20260928 12:58:46.970907 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41219
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab
--principal=oryx/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:47.058737 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:47.058969 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:47.058991 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:47.060212 18791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:47.060256 18791 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:58:47.060271 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:47.060283 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:47.060297 18791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:47.060309 18791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:47.061812 18791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.126.keytab
--principal=oryx/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:47.062021 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:47.062220 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:47.067013 18796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.067013 18799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.067085 18797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:47.067378 18791 server_base.cc:1034] running on GCE node
I20260928 12:58:47.067624 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:47.067914 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:47.069056 18791 hybrid_clock.cc:648] HybridClock initialized: now 1790600327069049 us; error 51 us; skew 500 ppm
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:47.078454 18791 init.cc:377] Logged in from keytab as oryx/127.11.11.126@KRBTEST.COM (short username oryx)
I20260928 12:58:47.078994 18791 webserver.cc:460] Webserver started at http://127.11.11.126:42443/ using document root <none> and password file <none>
I20260928 12:58:47.079243 18791 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:47.079291 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:47.079471 18791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:47.080713 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "9907617e03dd4daa91ede01d921a0b66"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "9889addb2381677cfb7e764f41919ca1"
server_key_iv: "f79af9d83d94a1886f5e4799a0c509da"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.081133 18791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "9907617e03dd4daa91ede01d921a0b66"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "9889addb2381677cfb7e764f41919ca1"
server_key_iv: "f79af9d83d94a1886f5e4799a0c509da"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.082551 18791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:58:47.083290 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.083475 18791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:47.083590 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "9907617e03dd4daa91ede01d921a0b66"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "9889addb2381677cfb7e764f41919ca1"
server_key_iv: "f79af9d83d94a1886f5e4799a0c509da"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.083666 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:47.105222 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:47.107316 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:47.107486 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:47.112259 18791 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41219
I20260928 12:58:47.112308 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41219 every 8 connection(s)
I20260928 12:58:47.112648 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:47.113299 18869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.115648 18869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Bootstrap starting.
I20260928 12:58:47.116245 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18791
I20260928 12:58:47.116353 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:47.116425 18869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.116595 11309 external_mini_cluster.cc:1428] Setting key b2a387f109ab4d56d1545c656bbbb68b
I20260928 12:58:47.116775 18869 log.cc:824] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:47.117547 18869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: No bootstrap required, opened a new log
I20260928 12:58:47.119601 18869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } }
I20260928 12:58:47.119777 18869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.119822 18869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9907617e03dd4daa91ede01d921a0b66, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.119944 18869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } }
I20260928 12:58:47.120038 18869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:47.120110 18869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:47.120159 18869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.120714 18869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } }
I20260928 12:58:47.120800 18869 leader_election.cc:302] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9907617e03dd4daa91ede01d921a0b66; no voters:
I20260928 12:58:47.120978 18869 leader_election.cc:288] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:47.121032 18874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:47.121172 18874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [term 1 LEADER]: Becoming Leader. State: Replica: 9907617e03dd4daa91ede01d921a0b66, State: Running, Role: LEADER
I20260928 12:58:47.121253 18869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:47.121268 18874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } }
I20260928 12:58:47.121604 18874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9907617e03dd4daa91ede01d921a0b66. Latest consensus state: current_term: 1 leader_uuid: "9907617e03dd4daa91ede01d921a0b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } } }
I20260928 12:58:47.121680 18874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:47.122138 18880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:47.122656 18880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:47.124394 18880 catalog_manager.cc:1347] Generated new cluster ID: e63f6f3389cb421d9b8a86faaa99c41b
I20260928 12:58:47.124446 18880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:47.125173 18875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9907617e03dd4daa91ede01d921a0b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9907617e03dd4daa91ede01d921a0b66" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41219 } } }
I20260928 12:58:47.125552 18875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66 [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:47.128391 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.117176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.11.11.126:41219)
0928 12:58:47.117367 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.117377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.117411 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:58:47.117914 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.117921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.117981 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.118147 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.118807 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.119716 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.120660 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.121336 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.121512 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.124663 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.124667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.124667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.124678 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.126637 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.127063 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.127066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.127066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.127169 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.127383 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.127384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.127384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.127688 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.127837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.128020 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.128184 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260928 12:58:47.139726 18880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:47.140399 18880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:47.144670 18880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Generated new TSK 0
I20260928 12:58:47.144912 18880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
I20260928 12:58:47.179622 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41219
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:47.263485 18895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:47.263725 18895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:47.263752 18895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:47.265064 18895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:47.265118 18895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:47.265165 18895 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:47.265185 18895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:47.266790 18895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:47.267059 18895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:47.267328 18895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:47.272490 18903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.272534 18901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:47.272676 18895 server_base.cc:1034] running on GCE node
W20260928 12:58:47.272490 18900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:47.272969 18895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:47.273198 18895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:47.274351 18895 hybrid_clock.cc:648] HybridClock initialized: now 1790600327274332 us; error 37 us; skew 500 ppm
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:47.277053 18895 init.cc:377] Logged in from keytab as oryx/127.11.11.65@KRBTEST.COM (short username oryx)
I20260928 12:58:47.277495 18895 webserver.cc:460] Webserver started at http://127.11.11.65:43413/ using document root <none> and password file <none>
I20260928 12:58:47.277763 18895 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:47.277833 18895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:47.277971 18895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:47.279011 18895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7c49aaa4f1a0b7fc8eabb2949d331fa6"
server_key_iv: "2ea1bad56406f07a026f9b956d44c7bb"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.279385 18895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7c49aaa4f1a0b7fc8eabb2949d331fa6"
server_key_iv: "2ea1bad56406f07a026f9b956d44c7bb"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.280808 18895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:47.281561 18910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.281737 18895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:47.281849 18895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7c49aaa4f1a0b7fc8eabb2949d331fa6"
server_key_iv: "2ea1bad56406f07a026f9b956d44c7bb"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.281929 18895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:47.298451 18895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:47.300989 18895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:47.301160 18895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:47.301716 18895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:47.301780 18895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.301826 18895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:47.301862 18895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.309270 18895 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:39101
I20260928 12:58:47.309320 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:39101 every 8 connection(s)
I20260928 12:58:47.309667 18895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:47.312721 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.302294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45617 (local address 127.11.11.126:41219)
0928 12:58:47.302363 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.302365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.302597 (+ 232us) server_negotiation.cc:407] Connection header received
0928 12:58:47.303383 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.303385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.303399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.303468 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.304473 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.305347 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.307132 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.307370 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.307425 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.310301 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.310303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.310303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.310310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.311471 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.311850 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.311851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.311851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.311905 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.312156 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.312156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.312156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.312275 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.312330 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.312573 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.312648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:58:47.312925 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.302266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.65:45617)
0928 12:58:47.302452 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.302505 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.303261 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.303575 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.303580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.304001 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.304366 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.304373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.305451 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.305452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.307031 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.307036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.307461 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.307463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.307636 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.308331 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.308343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.310201 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.311540 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.311547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.311549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.311783 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.311952 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.311952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.311953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.312089 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.312328 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.312334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.312456 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.312535 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.312682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260928 12:58:47.313431 19026 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41219
I20260928 12:58:47.313519 19026 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:47.313721 19026 heartbeater.cc:511] Master 127.11.11.126:41219 requested a full tablet report, sending...
I20260928 12:58:47.314314 18822 ts_manager.cc:194] Registered new tserver with Master: 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101)
I20260928 12:58:47.314713 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 18895
I20260928 12:58:47.314833 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:47.315057 11309 external_mini_cluster.cc:1428] Setting key 5663808edb8a9dd6a48198beb719358c
I20260928 12:58:47.315315 18822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:45617
I20260928 12:58:47.320719 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.314569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59519 (local address 127.11.11.126:41219)
0928 12:58:47.314628 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.314630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.314755 (+ 125us) server_negotiation.cc:407] Connection header received
0928 12:58:47.314807 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.314809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.314821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.314894 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.315207 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.315852 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.316988 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.317176 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.317255 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.318185 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.318187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.318188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.318195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.319574 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.319923 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.319924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.319925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.319981 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.320206 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.320207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.320207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.320317 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.320384 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.320551 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.320625 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:58:47.320739 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.314509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.65:59519)
0928 12:58:47.314681 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.314692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.314752 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.314898 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.314899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.315077 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.315146 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.315148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.315946 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.315947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.316893 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.316895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.317254 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.317255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.317325 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.317780 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.317785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.318095 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.319677 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.319678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.319679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.319855 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.320068 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.320069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.320070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.320129 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.320387 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.320387 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.320445 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.320523 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.320628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
I20260928 12:58:47.340600 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41219
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:47.423583 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:47.423796 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:47.423821 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:47.425107 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:47.425156 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:47.425179 19035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:47.425199 19035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:47.426688 19035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:47.426885 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:47.427067 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:47.432197 19041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.432183 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.432273 19040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:47.432377 19035 server_base.cc:1034] running on GCE node
I20260928 12:58:47.432621 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:47.432864 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:47.434032 19035 hybrid_clock.cc:648] HybridClock initialized: now 1790600327434010 us; error 30 us; skew 500 ppm
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:47.436614 19035 init.cc:377] Logged in from keytab as oryx/127.11.11.66@KRBTEST.COM (short username oryx)
I20260928 12:58:47.437086 19035 webserver.cc:460] Webserver started at http://127.11.11.66:44029/ using document root <none> and password file <none>
I20260928 12:58:47.437319 19035 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:47.437386 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:47.437522 19035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:47.438613 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7e3f3f3a7757d28e5b129e4a508f7af5"
server_key_iv: "b4f5908fe8bcdd27e6901ea4e2ece26f"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.438998 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7e3f3f3a7757d28e5b129e4a508f7af5"
server_key_iv: "b4f5908fe8bcdd27e6901ea4e2ece26f"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.440389 19035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:58:47.441124 19050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.441299 19035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:47.441397 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "7e3f3f3a7757d28e5b129e4a508f7af5"
server_key_iv: "b4f5908fe8bcdd27e6901ea4e2ece26f"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.441476 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:47.455227 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:47.457387 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:47.457556 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:47.458142 19035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:47.458201 19035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.458245 19035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:47.458272 19035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:47.465819 19035 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:41873
I20260928 12:58:47.466136 19165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:41873 every 8 connection(s)
I20260928 12:58:47.466226 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:58:47.468086 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.458574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49329 (local address 127.11.11.126:41219)
0928 12:58:47.458643 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.458646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.458893 (+ 247us) server_negotiation.cc:407] Connection header received
0928 12:58:47.459391 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.459393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.459406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.459509 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.460234 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.460743 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.461767 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.462007 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.462092 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.464969 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.464971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.464971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.464980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.466686 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.467073 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.467075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.467076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.467136 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.467401 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.467402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.467402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.467526 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.467594 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.467882 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.467964 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:58:47.468267 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.458559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.66:49329)
0928 12:58:47.458763 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.458804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.459297 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.459584 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.459587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.459886 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.460136 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.460141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.460838 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.460840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.461675 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.461677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.462105 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.462106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.462230 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.462790 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.462802 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.464871 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.466767 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.466771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.466773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.466998 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.467219 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.467220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.467221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.467331 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.467597 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.467604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.467749 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.467849 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.468015 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":125,"threads_started":1}
I20260928 12:58:47.468726 19166 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41219
I20260928 12:58:47.468811 19166 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:47.469022 19166 heartbeater.cc:511] Master 127.11.11.126:41219 requested a full tablet report, sending...
I20260928 12:58:47.469427 18822 ts_manager.cc:194] Registered new tserver with Master: 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:47.469866 18822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:49329
I20260928 12:58:47.475415 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.469885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:52641 (local address 127.11.11.126:41219)
0928 12:58:47.469931 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.469933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.470010 (+ 77us) server_negotiation.cc:407] Connection header received
0928 12:58:47.470085 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.470086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.470098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.470149 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.470863 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.471327 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.472007 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.472271 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.472328 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.473157 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.473158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.473158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.473164 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.474482 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.474772 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.474772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.474773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.474822 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.474978 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.474978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.474978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.475109 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.475171 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.475296 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.475352 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:47.475425 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.469802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.66:52641)
0928 12:58:47.469944 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.469957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.470025 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.470151 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.470152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.470734 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.470789 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.470791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.471418 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.471418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.471936 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.471937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.472336 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.472337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.472390 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.472785 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.472789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.473078 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.474547 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.474548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.474548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.474714 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.474877 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.474878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.474878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.474929 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.475169 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.475170 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.475226 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.475276 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.475355 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 12:58:47.475771 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19035
I20260928 12:58:47.475898 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:47.476142 11309 external_mini_cluster.cc:1428] Setting key 541515105d7df8a47138b4607aa550df
WARNING: no policy specified for oryx/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
I20260928 12:58:47.503098 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41219
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:47.584525 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:47.584800 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:47.584846 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:47.586097 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:47.586155 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:47.586187 19175 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:58:47.586239 19175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:47.587882 19175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.11.126:41219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:47.588160 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:47.588393 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:47.593461 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.593461 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:47.593587 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:47.593811 19175 server_base.cc:1034] running on GCE node
I20260928 12:58:47.594022 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:47.594275 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:47.595438 19175 hybrid_clock.cc:648] HybridClock initialized: now 1790600327595425 us; error 28 us; skew 500 ppm
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:47.597922 19175 init.cc:377] Logged in from keytab as oryx/127.11.11.67@KRBTEST.COM (short username oryx)
I20260928 12:58:47.598381 19175 webserver.cc:460] Webserver started at http://127.11.11.67:42025/ using document root <none> and password file <none>
I20260928 12:58:47.598652 19175 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:47.598724 19175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:47.598870 19175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:47.599956 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "469f316555e04010bdb315eb95cf0c0b"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "4cf609392f09f16621b6488a54573050"
server_key_iv: "5a1fe88cd44f60ab836df1baf12fe20a"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.600378 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "469f316555e04010bdb315eb95cf0c0b"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "4cf609392f09f16621b6488a54573050"
server_key_iv: "5a1fe88cd44f60ab836df1baf12fe20a"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.601749 19175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:47.602505 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.602660 19175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:47.602763 19175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "469f316555e04010bdb315eb95cf0c0b"
format_stamp: "Formatted at 2026-09-28 12:58:47 on dist-test-slave-8sjz"
server_key: "4cf609392f09f16621b6488a54573050"
server_key_iv: "5a1fe88cd44f60ab836df1baf12fe20a"
server_key_version: "encryptionkey@0"
I20260928 12:58:47.602857 19175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:47.619822 19175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:47.622016 19175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:47.622174 19175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:47.622798 19175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:47.622856 19175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.622902 19175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:47.622928 19175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.630290 19175 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:34877
I20260928 12:58:47.630381 19305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:34877 every 8 connection(s)
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:47.630734 19175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:47.633816 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.623199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:54327 (local address 127.11.11.126:41219)
0928 12:58:47.623241 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.623243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.623545 (+ 302us) server_negotiation.cc:407] Connection header received
0928 12:58:47.624201 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.624202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.624216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.624284 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.625329 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.626147 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.627879 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.628234 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.628311 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.631201 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.631202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.631203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.631210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.632696 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.633061 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.633062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.633062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.633112 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.633318 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.633319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.633319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.633424 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.633474 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.633693 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.633753 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:47.634013 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.623228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.67:54327)
0928 12:58:47.623380 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.623443 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.624092 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.624416 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.624421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.624856 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.625201 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.625209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.626246 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.626247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.627764 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.627767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.628325 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.628326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.628503 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.629084 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.629098 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.631105 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.632768 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.632773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.632774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.632997 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.633167 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.633167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.633168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.633269 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.633475 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.633479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.633595 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.633677 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.633803 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260928 12:58:47.634519 19306 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41219
I20260928 12:58:47.634618 19306 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:47.634800 19306 heartbeater.cc:511] Master 127.11.11.126:41219 requested a full tablet report, sending...
I20260928 12:58:47.635164 18822 ts_manager.cc:194] Registered new tserver with Master: 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877)
I20260928 12:58:47.635605 18822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:54327
I20260928 12:58:47.638332 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19175
I20260928 12:58:47.638434 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:47.638635 11309 external_mini_cluster.cc:1428] Setting key 66dc23130523db4c0b9c62a07e7d1a7a
I20260928 12:58:47.639683 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:58:47.641443 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.635587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:52173 (local address 127.11.11.126:41219)
0928 12:58:47.635629 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.635631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.635714 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:58:47.635799 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.635800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.635809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.635870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.636642 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.637092 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.637738 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.637909 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.637952 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.638973 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.638974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.638974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.638981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.640403 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.640799 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.640799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.640800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.640848 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.641027 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.641027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.641027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.641115 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.641170 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.641324 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.641381 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:47.641829 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.635509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.11.11.67:52173)
0928 12:58:47.635642 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.635655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.635740 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.635877 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.635878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.636523 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:47.636575 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.636577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.637170 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.637171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.637669 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.637670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.637964 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.637965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.638012 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.638493 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:47.638498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:47.638888 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:47.640475 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.640476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.640477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.640728 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.640903 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:47.640904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:47.640904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:47.640973 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:47.641172 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:47.641172 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:47.641222 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.641304 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.641426 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:58:47.661613 18873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:47.656148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.11.11.126:41219)
0928 12:58:47.656210 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.656213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.656224 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:47.656295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.656296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.656311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.656368 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.656958 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.657449 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.658190 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.658379 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.658452 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.661512 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55164: BlockingRecv error: recv got EOF from 127.0.0.1:55164 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 12:58:47 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:47.669705 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.662437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.11.11.126:41219)
0928 12:58:47.662518 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.662520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.662614 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:58:47.662707 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.662708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.662724 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.662786 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:47.663266 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.663756 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.664569 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.664761 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.664839 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.667001 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:47.667003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:47.667003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:47.667012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:47.668542 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.668942 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.668944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.668944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.669006 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:47.669214 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:47.669215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:47.669215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:47.669342 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:47.669428 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.669562 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.669630 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:58:47.671320 18822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:47.672961 18822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:47.679983 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.0.0.1:33808)
0928 12:58:47.675670 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.675718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.675793 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.676165 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.676168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.676171 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.676257 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.676259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677328 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.677329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.678776 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.678780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.679610 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.679611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.679714 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.679727 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.679805 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.679870 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":63,"thread_start_us":66,"threads_started":1}
I20260928 12:58:47.679991 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.11.11.65:39101)
0928 12:58:47.675840 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.675844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.675855 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:47.675906 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.675907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.675921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.676058 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.676347 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677195 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.678883 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.679486 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.679608 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.679703 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.679843 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.679887 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 12:58:47.680274 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:34877 (local address 127.0.0.1:34884)
0928 12:58:47.676023 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.676031 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.676090 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.676382 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.676382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.676385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.676432 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.676434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677662 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.677663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.678896 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.678899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.679599 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.679600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.679707 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.679726 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.679818 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.679870 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":9,"thread_start_us":96,"threads_started":1}
I20260928 12:58:47.680573 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.11.11.67:34877)
0928 12:58:47.675761 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.675767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.676163 (+ 396us) server_negotiation.cc:407] Connection header received
0928 12:58:47.676211 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.676214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.676232 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.676307 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.676499 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677561 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.678976 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.679484 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.679595 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.679686 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.679923 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.679973 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260928 12:58:47.680820 18960 tablet_service.cc:1467] Processing CreateTablet for tablet a8a1a5ead1124993947457cf995b4329 (DEFAULT_TABLE table=test-table [id=33f784dc5ba9427a87c513ee2fc1c512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.681190 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:41873 (local address 127.0.0.1:53476)
0928 12:58:47.675901 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.675909 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.675947 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.676227 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.676228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.676230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.676287 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.676288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677639 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.677641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.678786 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.678787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.680942 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.680944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.681022 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.681027 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.681080 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.681127 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":30,"thread_start_us":104,"threads_started":1}
I20260928 12:58:47.681262 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a1a5ead1124993947457cf995b4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.681576 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.675567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.11.11.66:41873)
0928 12:58:47.675845 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.675849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.675967 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:58:47.676016 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.676018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.676069 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.676145 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.676370 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.677525 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.679753 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.680804 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.681315 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.681431 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.681457 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.681491 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260928 12:58:47.682258 19100 tablet_service.cc:1467] Processing CreateTablet for tablet a8a1a5ead1124993947457cf995b4329 (DEFAULT_TABLE table=test-table [id=33f784dc5ba9427a87c513ee2fc1c512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.682598 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a1a5ead1124993947457cf995b4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.684876 19240 tablet_service.cc:1467] Processing CreateTablet for tablet a8a1a5ead1124993947457cf995b4329 (DEFAULT_TABLE table=test-table [id=33f784dc5ba9427a87c513ee2fc1c512]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.685127 19334 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Bootstrap starting.
I20260928 12:58:47.685286 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a1a5ead1124993947457cf995b4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.685772 19334 tablet_bootstrap.cc:654] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.686101 19334 log.cc:824] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:47.686793 19336 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Bootstrap starting.
I20260928 12:58:47.687127 19334 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: No bootstrap required, opened a new log
I20260928 12:58:47.687206 19334 ts_tablet_manager.cc:1397] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:58:47.687546 19336 tablet_bootstrap.cc:654] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.687872 19336 log.cc:824] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:47.688694 19336 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: No bootstrap required, opened a new log
I20260928 12:58:47.688767 19336 ts_tablet_manager.cc:1397] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 12:58:47.689414 19338 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Bootstrap starting.
I20260928 12:58:47.689661 19334 raft_consensus.cc:348] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.689834 19334 raft_consensus.cc:374] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.689870 19334 raft_consensus.cc:729] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.690012 19334 consensus_queue.cc:260] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.690080 19338 tablet_bootstrap.cc:654] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.690289 19334 ts_tablet_manager.cc:1428] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:58:47.690447 19026 heartbeater.cc:503] Master 127.11.11.126:41219 was elected leader, sending a full tablet report...
I20260928 12:58:47.690436 19338 log.cc:824] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:47.690824 19336 raft_consensus.cc:348] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.690991 19336 raft_consensus.cc:374] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.691075 19336 raft_consensus.cc:729] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.691262 19336 consensus_queue.cc:260] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.691534 19338 tablet_bootstrap.cc:492] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: No bootstrap required, opened a new log
I20260928 12:58:47.691586 19336 ts_tablet_manager.cc:1428] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:58:47.691738 19166 heartbeater.cc:503] Master 127.11.11.126:41219 was elected leader, sending a full tablet report...
I20260928 12:58:47.692028 19338 ts_tablet_manager.cc:1397] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:58:47.693936 19338 raft_consensus.cc:348] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.694080 19338 raft_consensus.cc:374] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.694104 19338 raft_consensus.cc:729] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.694237 19338 consensus_queue.cc:260] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.694531 19338 ts_tablet_manager.cc:1428] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:58:47.694567 19306 heartbeater.cc:503] Master 127.11.11.126:41219 was elected leader, sending a full tablet report...
W20260928 12:58:47.716846 19167 tablet.cc:2369] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:47.728683 19341 raft_consensus.cc:482] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:47.728817 19341 raft_consensus.cc:504] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.729250 19341 leader_election.cc:288] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877), 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101)
I20260928 12:58:47.732195 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.729476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:34877 (local address 127.11.11.66:55747)
0928 12:58:47.729681 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.729691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.729758 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.729907 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.729908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.729909 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.730008 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.730010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.730637 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.730637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.731406 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.731408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.731933 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.732024 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.732028 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.732090 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.732121 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260928 12:58:47.732247 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.729558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:55747 (local address 127.11.11.67:34877)
0928 12:58:47.729622 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.729624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.729755 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:58:47.729832 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.729833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.729842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.729897 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.730085 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.730534 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731501 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.731855 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731911 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.731936 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.732110 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.732158 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:58:47.732570 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a1a5ead1124993947457cf995b4329" candidate_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b" is_pre_election: true
I20260928 12:58:47.732640 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.729460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.11.11.66:60393)
0928 12:58:47.729520 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.729534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.729582 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.729823 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.729824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.729826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.729907 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.729909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.730594 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.730595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.731402 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.731404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.731967 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.732472 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.732476 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.732538 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.732580 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:58:47.732656 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.729598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:60393 (local address 127.11.11.65:39101)
0928 12:58:47.729661 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.729664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.729675 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:47.729734 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.729735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.729746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.729808 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.729991 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.730509 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731515 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.731869 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.731955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.731964 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.732557 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.732606 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:58:47.732750 19260 raft_consensus.cc:2454] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2e7270ac344d01bc12e1ee9e0c79d7 in term 0.
I20260928 12:58:47.732941 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a1a5ead1124993947457cf995b4329" candidate_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" is_pre_election: true
I20260928 12:58:47.732977 19051 leader_election.cc:302] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469f316555e04010bdb315eb95cf0c0b, 4f2e7270ac344d01bc12e1ee9e0c79d7; no voters:
I20260928 12:58:47.733093 19341 raft_consensus.cc:2781] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:47.733103 18979 raft_consensus.cc:2454] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2e7270ac344d01bc12e1ee9e0c79d7 in term 0.
I20260928 12:58:47.733148 19341 raft_consensus.cc:482] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:47.733174 19341 raft_consensus.cc:3037] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.734213 19341 raft_consensus.cc:504] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.734388 19341 leader_election.cc:288] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [CANDIDATE]: Term 1 election: Requested vote from peers 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877), 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101)
I20260928 12:58:47.734526 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a1a5ead1124993947457cf995b4329" candidate_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b"
I20260928 12:58:47.734601 19260 raft_consensus.cc:3037] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.734818 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a1a5ead1124993947457cf995b4329" candidate_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
I20260928 12:58:47.734911 18979 raft_consensus.cc:3037] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.735554 19260 raft_consensus.cc:2454] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2e7270ac344d01bc12e1ee9e0c79d7 in term 1.
I20260928 12:58:47.735709 19051 leader_election.cc:302] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469f316555e04010bdb315eb95cf0c0b, 4f2e7270ac344d01bc12e1ee9e0c79d7; no voters:
I20260928 12:58:47.735769 18979 raft_consensus.cc:2454] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2e7270ac344d01bc12e1ee9e0c79d7 in term 1.
I20260928 12:58:47.735790 19341 raft_consensus.cc:2781] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:47.735965 19341 raft_consensus.cc:686] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Running, Role: LEADER
I20260928 12:58:47.736089 19341 consensus_queue.cc:237] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } }
I20260928 12:58:47.736912 18822 catalog_manager.cc:5495] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66). New cstate: current_term: 1 leader_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:47.753495 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.751127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.11.11.66:41873)
0928 12:58:47.751199 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.751201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.751209 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:47.751243 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.751244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.751262 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.751323 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:47.751445 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.751915 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.752694 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.752876 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.752930 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.753131 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:47.753238 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:47.753299 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.753376 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.753396 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:58:47.755354 18979 raft_consensus.cc:1261] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Refusing update from remote peer 4f2e7270ac344d01bc12e1ee9e0c79d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:47.755363 19260 raft_consensus.cc:1261] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Refusing update from remote peer 4f2e7270ac344d01bc12e1ee9e0c79d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:47.755771 19341 consensus_queue.cc:1035] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:47.755914 19344 consensus_queue.cc:1035] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:47.759508 19348 mvcc.cc:205] Tried to move back new op lower bound from 7334298942483218432 to 7334298942407651328. Current Snapshot: MvccSnapshot[applied={T|T < 7334298942483218432}]
I20260928 12:58:47.760702 18822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:47.760807 18822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:47.761618 18822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Sending DeleteTablet for 3 replicas of tablet a8a1a5ead1124993947457cf995b4329
I20260928 12:58:47.761960 19240 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a1a5ead1124993947457cf995b4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:47 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34884
I20260928 12:58:47.762086 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a1a5ead1124993947457cf995b4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:47 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53476
I20260928 12:58:47.762213 18960 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a1a5ead1124993947457cf995b4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:47 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:33808
I20260928 12:58:47.762238 18822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:47.762401 18822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:47.762463 19359 tablet_replica.cc:330] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: stopping tablet replica
I20260928 12:58:47.762559 19359 raft_consensus.cc:2229] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:47.762810 19357 tablet_replica.cc:330] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: stopping tablet replica
I20260928 12:58:47.762945 19359 raft_consensus.cc:2258] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:47.762960 19357 raft_consensus.cc:2229] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:47.763168 19357 raft_consensus.cc:2258] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:47.763455 19359 ts_tablet_manager.cc:1905] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:47.763720 19357 ts_tablet_manager.cc:1905] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:47.763849 19100 tablet_service.cc:1467] Processing CreateTablet for tablet b87914bab42b4c62aea104802b50265f (DEFAULT_TABLE table=test-table [id=21675764b1ff48ef887f4b5e6b789ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.763990 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87914bab42b4c62aea104802b50265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.765342 19359 ts_tablet_manager.cc:1918] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:47.765427 19359 log.cc:1196] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/a8a1a5ead1124993947457cf995b4329
I20260928 12:58:47.765528 19336 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Bootstrap starting.
I20260928 12:58:47.765591 19240 tablet_service.cc:1467] Processing CreateTablet for tablet b87914bab42b4c62aea104802b50265f (DEFAULT_TABLE table=test-table [id=21675764b1ff48ef887f4b5e6b789ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.765697 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87914bab42b4c62aea104802b50265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.765717 19359 ts_tablet_manager.cc:1939] T a8a1a5ead1124993947457cf995b4329 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting consensus metadata
I20260928 12:58:47.765978 19336 tablet_bootstrap.cc:654] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.766350 18810 catalog_manager.cc:4841] TS 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101): tablet a8a1a5ead1124993947457cf995b4329 (table test-table [id=33f784dc5ba9427a87c513ee2fc1c512]) successfully deleted
I20260928 12:58:47.766554 18960 tablet_service.cc:1467] Processing CreateTablet for tablet b87914bab42b4c62aea104802b50265f (DEFAULT_TABLE table=test-table [id=21675764b1ff48ef887f4b5e6b789ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:47.766669 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87914bab42b4c62aea104802b50265f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:47.766677 19336 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: No bootstrap required, opened a new log
I20260928 12:58:47.766726 19336 ts_tablet_manager.cc:1397] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:47.766885 19336 raft_consensus.cc:348] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.766966 19336 raft_consensus.cc:374] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.766992 19336 raft_consensus.cc:729] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.767035 19336 consensus_queue.cc:260] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.767128 19336 ts_tablet_manager.cc:1428] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:47.767481 19357 ts_tablet_manager.cc:1918] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:47.767601 19357 log.cc:1196] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/a8a1a5ead1124993947457cf995b4329
I20260928 12:58:47.767869 19334 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Bootstrap starting.
I20260928 12:58:47.767976 19357 ts_tablet_manager.cc:1939] T a8a1a5ead1124993947457cf995b4329 P 469f316555e04010bdb315eb95cf0c0b: Deleting consensus metadata
I20260928 12:58:47.768299 19334 tablet_bootstrap.cc:654] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.768546 18807 catalog_manager.cc:4841] TS 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877): tablet a8a1a5ead1124993947457cf995b4329 (table test-table [id=33f784dc5ba9427a87c513ee2fc1c512]) successfully deleted
I20260928 12:58:47.768719 19358 tablet_replica.cc:330] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: stopping tablet replica
I20260928 12:58:47.768860 19358 raft_consensus.cc:2229] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:47.768970 19358 raft_consensus.cc:2258] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:47.769316 19338 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Bootstrap starting.
I20260928 12:58:47.769378 19358 ts_tablet_manager.cc:1905] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:47.769721 19338 tablet_bootstrap.cc:654] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:47.770341 19338 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: No bootstrap required, opened a new log
I20260928 12:58:47.770387 19338 ts_tablet_manager.cc:1397] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:47.770522 19338 raft_consensus.cc:348] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.770584 19338 raft_consensus.cc:374] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.770603 19338 raft_consensus.cc:729] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.770658 19338 consensus_queue.cc:260] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.770700 19334 tablet_bootstrap.cc:492] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: No bootstrap required, opened a new log
I20260928 12:58:47.770743 19334 ts_tablet_manager.cc:1397] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 12:58:47.770754 19338 ts_tablet_manager.cc:1428] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:47.770895 19334 raft_consensus.cc:348] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.770975 19334 raft_consensus.cc:374] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:47.771004 19334 raft_consensus.cc:729] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Initialized, Role: FOLLOWER
I20260928 12:58:47.771046 19334 consensus_queue.cc:260] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.771130 19334 ts_tablet_manager.cc:1428] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:47.771301 19358 ts_tablet_manager.cc:1918] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:47.771373 19358 log.cc:1196] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/a8a1a5ead1124993947457cf995b4329
I20260928 12:58:47.771646 19358 ts_tablet_manager.cc:1939] T a8a1a5ead1124993947457cf995b4329 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting consensus metadata
I20260928 12:58:47.772058 18808 catalog_manager.cc:4841] TS 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873): tablet a8a1a5ead1124993947457cf995b4329 (table test-table [id=33f784dc5ba9427a87c513ee2fc1c512]) successfully deleted
W20260928 12:58:47.810469 19027 tablet.cc:2369] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:58:47.881331 19307 tablet.cc:2369] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:58:47.957636 19342 raft_consensus.cc:482] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:47.957757 19342 raft_consensus.cc:504] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.958225 19342 leader_election.cc:288] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:47.961103 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.958428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:41873 (local address 127.11.11.67:47763)
0928 12:58:47.958486 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.958501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.958566 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.958729 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.958730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.958732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.958814 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.958816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.959460 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.959461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.960287 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.960289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.960837 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.960914 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.960918 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.960981 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.961028 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:58:47.961103 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.958513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:47763 (local address 127.11.11.66:41873)
0928 12:58:47.958568 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.958570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.958580 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:47.958637 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.958638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.958651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.958725 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.958888 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.959370 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960376 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.960757 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.960851 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.960988 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.961028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:58:47.961272 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.958428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.11.11.67:39235)
0928 12:58:47.958627 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:47.958639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:47.958750 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:47.958895 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:47.958896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:47.958897 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:47.958987 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.958988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.959640 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.959641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.960411 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:47.960413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.960994 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:47.961067 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.961071 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:58:47.961136 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:58:47.961176 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":66,"thread_start_us":81,"threads_started":1}
I20260928 12:58:47.961253 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:47.958541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:39235 (local address 127.11.11.65:39101)
0928 12:58:47.958634 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:47.958636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:47.958739 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:58:47.958827 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:47.958827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:47.958836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:47.958893 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:47.959059 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.959526 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960502 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:47.960863 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:47.960953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:47.960963 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:47.961141 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 12:58:47.961184 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 12:58:47.961513 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87914bab42b4c62aea104802b50265f" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" is_pre_election: true
I20260928 12:58:47.961539 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87914bab42b4c62aea104802b50265f" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" is_pre_election: true
I20260928 12:58:47.961653 18979 raft_consensus.cc:2454] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 0.
I20260928 12:58:47.961683 19120 raft_consensus.cc:2454] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 0.
I20260928 12:58:47.961840 19194 leader_election.cc:302] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:47.961987 19342 raft_consensus.cc:2781] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:47.962030 19342 raft_consensus.cc:482] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:47.962045 19342 raft_consensus.cc:3037] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.962776 19342 raft_consensus.cc:504] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.962932 19342 leader_election.cc:288] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 election: Requested vote from peers 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:47.963085 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87914bab42b4c62aea104802b50265f" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
I20260928 12:58:47.963171 18979 raft_consensus.cc:3037] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.963116 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87914bab42b4c62aea104802b50265f" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
I20260928 12:58:47.963222 19120 raft_consensus.cc:3037] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:47.963801 18979 raft_consensus.cc:2454] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 1.
I20260928 12:58:47.963969 19120 raft_consensus.cc:2454] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 1.
I20260928 12:58:47.963973 19194 leader_election.cc:302] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:47.964174 19342 raft_consensus.cc:2781] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:47.964298 19342 raft_consensus.cc:686] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 1 LEADER]: Becoming Leader. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Running, Role: LEADER
I20260928 12:58:47.964375 19342 consensus_queue.cc:237] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:47.965092 18823 catalog_manager.cc:5495] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 469f316555e04010bdb315eb95cf0c0b (127.11.11.67). New cstate: current_term: 1 leader_uuid: "469f316555e04010bdb315eb95cf0c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } health_report { overall_health: HEALTHY } } }
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600327, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.126@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:48.032269 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.024919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.11.11.126:41219)
0928 12:58:48.024974 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.024976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.025138 (+ 162us) server_negotiation.cc:407] Connection header received
0928 12:58:48.025254 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.025255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.025266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.025335 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:48.025823 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.026309 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.027026 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.027210 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.027285 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.029569 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:48.029570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:48.029570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:48.029578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:48.031053 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:48.031455 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:48.031456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:48.031457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:48.031515 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:48.031703 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:48.031704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:48.031705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:48.031820 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:48.031892 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.032068 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.032155 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:58:48.032356 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.024850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41219 (local address 127.0.0.1:55176)
0928 12:58:48.025062 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.025072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.025160 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.025340 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.025341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.025676 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:48.025729 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.025732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.026403 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.026404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.026935 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.026936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.027282 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.027283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.027334 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.027903 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:48.027907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:48.029474 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:48.031166 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:48.031174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:48.031177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:48.031384 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:48.031577 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:48.031578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:48.031578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:48.031642 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:48.031882 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:48.031889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:48.031936 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.032017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.032232 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":71,"thread_start_us":130,"threads_started":1}
I20260928 12:58:48.034421 18823 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:55176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:58:48.034633 18823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:48.036023 18960 tablet_service.cc:1467] Processing CreateTablet for tablet 1213a7b6172d4fed901c21cfea0c94e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=840980a421c04ac193a9fa0137d0e71a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:48.036022 19240 tablet_service.cc:1467] Processing CreateTablet for tablet 1213a7b6172d4fed901c21cfea0c94e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=840980a421c04ac193a9fa0137d0e71a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:48.036078 19100 tablet_service.cc:1467] Processing CreateTablet for tablet 1213a7b6172d4fed901c21cfea0c94e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=840980a421c04ac193a9fa0137d0e71a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:58:48.036223 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1213a7b6172d4fed901c21cfea0c94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.036223 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1213a7b6172d4fed901c21cfea0c94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.036250 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1213a7b6172d4fed901c21cfea0c94e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.037393 19336 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Bootstrap starting.
I20260928 12:58:48.037753 19334 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: Bootstrap starting.
I20260928 12:58:48.037748 19338 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: Bootstrap starting.
I20260928 12:58:48.037786 19336 tablet_bootstrap.cc:654] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.038159 19334 tablet_bootstrap.cc:654] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.038174 19338 tablet_bootstrap.cc:654] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.038570 19336 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: No bootstrap required, opened a new log
I20260928 12:58:48.038631 19336 ts_tablet_manager.cc:1397] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.038762 19336 raft_consensus.cc:348] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.038822 19338 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: No bootstrap required, opened a new log
I20260928 12:58:48.038859 19336 raft_consensus.cc:374] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.038872 19338 ts_tablet_manager.cc:1397] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.038878 19334 tablet_bootstrap.cc:492] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: No bootstrap required, opened a new log
I20260928 12:58:48.038887 19336 raft_consensus.cc:729] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.038921 19334 ts_tablet_manager.cc:1397] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.038932 19336 consensus_queue.cc:260] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.039029 19336 ts_tablet_manager.cc:1428] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:48.039042 19338 raft_consensus.cc:348] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.039064 19334 raft_consensus.cc:348] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.039119 19338 raft_consensus.cc:374] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.039125 19334 raft_consensus.cc:374] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.039149 19334 raft_consensus.cc:729] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.039149 19338 raft_consensus.cc:729] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.039153 19354 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.039193 19338 consensus_queue.cc:260] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.039204 19334 consensus_queue.cc:260] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.039232 19354 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.039287 19338 ts_tablet_manager.cc:1428] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:48.039296 19334 ts_tablet_manager.cc:1428] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:58:48.039420 19339 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.039464 19342 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.039520 19339 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.039543 19342 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.115680 19339 raft_consensus.cc:482] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:48.115783 19339 raft_consensus.cc:504] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.116089 19339 leader_election.cc:288] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:48.119068 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.116374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:40041 (local address 127.11.11.66:41873)
0928 12:58:48.116427 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.116429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.116499 (+ 70us) server_negotiation.cc:407] Connection header received
0928 12:58:48.116573 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.116574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.116584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.116643 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:48.116817 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.117320 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118344 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.118715 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.118782 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.118936 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.118987 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:58:48.119093 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.116367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59367 (local address 127.11.11.67:34877)
0928 12:58:48.116432 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.116434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.116541 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:58:48.116600 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.116601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.116615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.116689 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:58:48.116875 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.117323 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118346 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.118733 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118820 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.118829 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.118993 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.119032 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:58:48.119091 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.116282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:41873 (local address 127.11.11.65:40041)
0928 12:58:48.116416 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.116430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.116499 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.116658 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.116659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.116662 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:48.116747 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.116749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.117409 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.117410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.118248 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.118250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.118794 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.118857 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.118862 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.118912 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.118945 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260928 12:58:48.119112 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.116282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:34877 (local address 127.11.11.65:59367)
0928 12:58:48.116448 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.116459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.116541 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.116711 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.116712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.116714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:58:48.116796 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.116798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.117416 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.117417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.118246 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.118249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.118832 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.118833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.118910 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.118915 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.118973 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.119010 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":42,"thread_start_us":67,"threads_started":1}
I20260928 12:58:48.119392 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1213a7b6172d4fed901c21cfea0c94e7" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" is_pre_election: true
I20260928 12:58:48.119524 19120 raft_consensus.cc:2454] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 0.
I20260928 12:58:48.119613 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1213a7b6172d4fed901c21cfea0c94e7" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b" is_pre_election: true
I20260928 12:58:48.119696 19260 raft_consensus.cc:2454] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 0.
I20260928 12:58:48.119876 18911 leader_election.cc:302] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:48.120064 19339 raft_consensus.cc:2781] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:48.120110 19339 raft_consensus.cc:482] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:48.120141 19339 raft_consensus.cc:3037] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.120995 19339 raft_consensus.cc:504] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.121155 19339 leader_election.cc:288] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 election: Requested vote from peers 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:48.121347 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1213a7b6172d4fed901c21cfea0c94e7" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
I20260928 12:58:48.121347 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1213a7b6172d4fed901c21cfea0c94e7" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b"
I20260928 12:58:48.121433 19260 raft_consensus.cc:3037] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.121433 19120 raft_consensus.cc:3037] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.122259 19120 raft_consensus.cc:2454] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 1.
I20260928 12:58:48.122259 19260 raft_consensus.cc:2454] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 1.
I20260928 12:58:48.122423 18911 leader_election.cc:302] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:48.122519 19339 raft_consensus.cc:2781] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:48.122651 19339 raft_consensus.cc:686] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 LEADER]: Becoming Leader. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Running, Role: LEADER
I20260928 12:58:48.122733 19339 consensus_queue.cc:237] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } }
I20260928 12:58:48.123127 19377 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: TxnStatusTablet state changed. Reason: New leader 27f3496526ce4233a7c0f4070e3b5b6e. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.123241 19377 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:48.123430 19379 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:58:48.123553 19379 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:58:48.123533 18822 catalog_manager.cc:5495] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65). New cstate: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:48.136546 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.133311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.11.11.65:39101)
0928 12:58:48.133383 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.133385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.133395 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:48.133493 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.133494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.133512 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.133564 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.133716 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.134603 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.135736 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.135919 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.135973 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.136224 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.136326 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.136385 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.136462 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.136481 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:58:48.136601 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.133251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.0.0.1:33822)
0928 12:58:48.133344 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.133356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.133402 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.133583 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.133584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.133586 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:48.133645 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.133647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.134687 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.134688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.135650 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.135653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.135993 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.135994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.136091 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.136116 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.136394 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.136403 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.136460 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.136513 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 12:58:48.137759 19260 raft_consensus.cc:1261] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Refusing update from remote peer 27f3496526ce4233a7c0f4070e3b5b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.137861 19120 raft_consensus.cc:1261] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Refusing update from remote peer 27f3496526ce4233a7c0f4070e3b5b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.138118 19377 consensus_queue.cc:1035] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.138298 19339 consensus_queue.cc:1035] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.138975 19354 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: TxnStatusTablet state changed. Reason: New leader 27f3496526ce4233a7c0f4070e3b5b6e. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.139071 19354 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.139223 19342 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: TxnStatusTablet state changed. Reason: New leader 27f3496526ce4233a7c0f4070e3b5b6e. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.139307 19342 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.140095 19339 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.140196 19339 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:48.140342 19349 mvcc.cc:205] Tried to move back new op lower bound from 7334298944050114560 to 7334298943991263232. Current Snapshot: MvccSnapshot[applied={T|T < 7334298944050114560}]
I20260928 12:58:48.140440 19342 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.140470 19339 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.140517 19342 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 469f316555e04010bdb315eb95cf0c0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.140535 19339 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 27f3496526ce4233a7c0f4070e3b5b6e: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:58:48.140928 19347 mvcc.cc:205] Tried to move back new op lower bound from 7334298944050114560 to 7334298943991263232. Current Snapshot: MvccSnapshot[applied={T|T < 7334298944050114560 or (T in {7334298944050114560})}]
I20260928 12:58:48.141026 19354 tablet_replica.cc:439] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } }
I20260928 12:58:48.141090 19354 tablet_replica.cc:442] T 1213a7b6172d4fed901c21cfea0c94e7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:58:48.144667 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.141905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.11.11.67:34877)
0928 12:58:48.141944 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.141946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.142044 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:58:48.142084 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.142085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.142108 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.142175 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.142286 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.143031 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.143797 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.144136 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.144222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.144342 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.144475 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.144550 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.144567 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.144592 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:58:48.149809 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.147078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.11.11.67:54597)
0928 12:58:48.147217 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.147228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.147286 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.147418 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.147419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.147420 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:48.147461 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.147462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.148344 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.148345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.148916 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.148918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.149397 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.149397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.149442 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.149456 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.149649 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.149651 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.149700 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.149740 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260928 12:58:48.149814 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.147149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:54597 (local address 127.11.11.65:39101)
0928 12:58:48.147201 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.147203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.147278 (+ 75us) server_negotiation.cc:407] Connection header received
0928 12:58:48.147349 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.147349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.147359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.147417 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.147500 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.148277 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.149000 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.149317 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.149382 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.149532 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.149566 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.149623 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.149714 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.149736 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:58:48.152345 18979 raft_consensus.cc:1261] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Refusing update from remote peer 469f316555e04010bdb315eb95cf0c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.152554 19120 raft_consensus.cc:1261] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Refusing update from remote peer 469f316555e04010bdb315eb95cf0c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.152650 19342 consensus_queue.cc:1035] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.152974 19342 consensus_queue.cc:1035] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.162567 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.159385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:34877 (local address 127.11.11.65:47167)
0928 12:58:48.159543 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.159556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.159607 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.159740 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.159741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.159742 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:48.159800 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.159801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.160875 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.160876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.161640 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.161643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.162119 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.162120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.162167 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.162183 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.162375 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.162377 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.162430 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.162471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 12:58:48.162544 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.159515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47167 (local address 127.11.11.67:34877)
0928 12:58:48.159583 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.159585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.159596 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:48.159661 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.159662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.159672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.159739 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.159839 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.160681 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.161716 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.162021 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.162095 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.162289 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.162325 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.162377 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.162437 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.162462 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:58:48.171928 18821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.172073 18821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.172735 18821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Sending DeleteTablet for 3 replicas of tablet b87914bab42b4c62aea104802b50265f
I20260928 12:58:48.173069 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet b87914bab42b4c62aea104802b50265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53476
I20260928 12:58:48.173177 19358 tablet_replica.cc:330] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: stopping tablet replica
I20260928 12:58:48.173146 18960 tablet_service.cc:1514] Processing DeleteTablet for tablet b87914bab42b4c62aea104802b50265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:33808
I20260928 12:58:48.173230 19358 raft_consensus.cc:2229] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.173269 19358 raft_consensus.cc:2258] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.173348 19359 tablet_replica.cc:330] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: stopping tablet replica
I20260928 12:58:48.173404 19359 raft_consensus.cc:2229] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.173442 19359 raft_consensus.cc:2258] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.173784 19358 ts_tablet_manager.cc:1905] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.173790 19359 ts_tablet_manager.cc:1905] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.175565 19240 tablet_service.cc:1514] Processing DeleteTablet for tablet b87914bab42b4c62aea104802b50265f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34884
I20260928 12:58:48.175683 19357 tablet_replica.cc:330] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: stopping tablet replica
I20260928 12:58:48.175732 19357 raft_consensus.cc:2229] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:48.175828 19357 raft_consensus.cc:2258] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.175859 19359 ts_tablet_manager.cc:1918] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:48.175913 19359 log.cc:1196] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/b87914bab42b4c62aea104802b50265f
I20260928 12:58:48.176200 19359 ts_tablet_manager.cc:1939] T b87914bab42b4c62aea104802b50265f P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting consensus metadata
I20260928 12:58:48.176227 19357 ts_tablet_manager.cc:1905] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.176522 18810 catalog_manager.cc:4841] TS 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101): tablet b87914bab42b4c62aea104802b50265f (table test-table [id=21675764b1ff48ef887f4b5e6b789ca6]) successfully deleted
I20260928 12:58:48.177767 19357 ts_tablet_manager.cc:1918] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:48.177832 19357 log.cc:1196] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/b87914bab42b4c62aea104802b50265f
I20260928 12:58:48.178059 19357 ts_tablet_manager.cc:1939] T b87914bab42b4c62aea104802b50265f P 469f316555e04010bdb315eb95cf0c0b: Deleting consensus metadata
I20260928 12:58:48.178586 18807 catalog_manager.cc:4841] TS 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877): tablet b87914bab42b4c62aea104802b50265f (table test-table [id=21675764b1ff48ef887f4b5e6b789ca6]) successfully deleted
I20260928 12:58:48.181439 19358 ts_tablet_manager.cc:1918] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:48.181524 19358 log.cc:1196] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/b87914bab42b4c62aea104802b50265f
I20260928 12:58:48.181771 19358 ts_tablet_manager.cc:1939] T b87914bab42b4c62aea104802b50265f P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting consensus metadata
I20260928 12:58:48.182137 18808 catalog_manager.cc:4841] TS 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873): tablet b87914bab42b4c62aea104802b50265f (table test-table [id=21675764b1ff48ef887f4b5e6b789ca6]) successfully deleted
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[18778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:58:48.211342 18873 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:48.204117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.11.11.126:41219)
0928 12:58:48.204172 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.204175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.204365 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:58:48.204438 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.204439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.204453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.204520 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:48.205366 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.206231 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.207336 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.207656 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.207724 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.211271 (+ 3547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55186: BlockingRecv error: recv got EOF from 127.0.0.1:55186 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[18778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM
I20260928 12:58:48.221808 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.212585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.11.11.126:41219)
0928 12:58:48.212625 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.212627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.212728 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:58:48.212819 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.212821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.212834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.212901 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:48.213554 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.214357 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.215434 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.215715 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.215794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.218567 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:48.218568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:48.218568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:48.218577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:48.220427 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:48.220848 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:48.220849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:48.220850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:48.220969 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:48.221194 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:48.221195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:48.221195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:48.221351 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:48.221457 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.221623 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.221712 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:58:48.223726 18821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:48.223922 18821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:48.225706 18960 tablet_service.cc:1467] Processing CreateTablet for tablet a249a5ed779e442baa85a8960af003d7 (DEFAULT_TABLE table=test-table [id=371eb9e033aa4a6dbea1179c0da31d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.225852 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a249a5ed779e442baa85a8960af003d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.227227 19334 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Bootstrap starting.
I20260928 12:58:48.227246 19100 tablet_service.cc:1467] Processing CreateTablet for tablet a249a5ed779e442baa85a8960af003d7 (DEFAULT_TABLE table=test-table [id=371eb9e033aa4a6dbea1179c0da31d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.227367 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a249a5ed779e442baa85a8960af003d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.227699 19334 tablet_bootstrap.cc:654] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.228787 19336 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Bootstrap starting.
I20260928 12:58:48.228895 19334 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: No bootstrap required, opened a new log
I20260928 12:58:48.228942 19334 ts_tablet_manager.cc:1397] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:58:48.229156 19336 tablet_bootstrap.cc:654] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.229104 19334 raft_consensus.cc:348] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.229197 19334 raft_consensus.cc:374] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.229221 19334 raft_consensus.cc:729] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.229270 19334 consensus_queue.cc:260] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.229378 19334 ts_tablet_manager.cc:1428] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:48.229491 19240 tablet_service.cc:1467] Processing CreateTablet for tablet a249a5ed779e442baa85a8960af003d7 (DEFAULT_TABLE table=test-table [id=371eb9e033aa4a6dbea1179c0da31d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.229624 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a249a5ed779e442baa85a8960af003d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.230917 19338 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Bootstrap starting.
I20260928 12:58:48.231356 19338 tablet_bootstrap.cc:654] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.231640 19336 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: No bootstrap required, opened a new log
I20260928 12:58:48.231686 19336 ts_tablet_manager.cc:1397] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 12:58:48.231796 19336 raft_consensus.cc:348] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.231861 19336 raft_consensus.cc:374] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.231880 19336 raft_consensus.cc:729] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.231920 19336 consensus_queue.cc:260] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.232009 19336 ts_tablet_manager.cc:1428] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:48.233145 19338 tablet_bootstrap.cc:492] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: No bootstrap required, opened a new log
I20260928 12:58:48.233214 19338 ts_tablet_manager.cc:1397] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:58:48.233359 19338 raft_consensus.cc:348] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.233428 19338 raft_consensus.cc:374] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.233449 19338 raft_consensus.cc:729] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.233503 19338 consensus_queue.cc:260] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.233592 19338 ts_tablet_manager.cc:1428] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:48.531589 19342 raft_consensus.cc:482] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:48.531740 19342 raft_consensus.cc:504] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.531948 19342 leader_election.cc:288] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:48.532244 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a249a5ed779e442baa85a8960af003d7" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" is_pre_election: true
I20260928 12:58:48.532284 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a249a5ed779e442baa85a8960af003d7" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" is_pre_election: true
I20260928 12:58:48.532347 18979 raft_consensus.cc:2454] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 0.
I20260928 12:58:48.532387 19120 raft_consensus.cc:2454] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 0.
I20260928 12:58:48.532516 19194 leader_election.cc:302] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:48.532621 19342 raft_consensus.cc:2781] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:48.532677 19342 raft_consensus.cc:482] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:48.532693 19342 raft_consensus.cc:3037] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.533339 19342 raft_consensus.cc:504] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.533473 19342 leader_election.cc:288] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 election: Requested vote from peers 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101), 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873)
I20260928 12:58:48.533607 18979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a249a5ed779e442baa85a8960af003d7" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3496526ce4233a7c0f4070e3b5b6e"
I20260928 12:58:48.533622 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a249a5ed779e442baa85a8960af003d7" candidate_uuid: "469f316555e04010bdb315eb95cf0c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
I20260928 12:58:48.533702 18979 raft_consensus.cc:3037] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.533701 19120 raft_consensus.cc:3037] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.534515 18979 raft_consensus.cc:2454] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 1.
I20260928 12:58:48.534514 19120 raft_consensus.cc:2454] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469f316555e04010bdb315eb95cf0c0b in term 1.
I20260928 12:58:48.534718 19194 leader_election.cc:302] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b, 4f2e7270ac344d01bc12e1ee9e0c79d7; no voters:
I20260928 12:58:48.534816 19342 raft_consensus.cc:2781] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:48.534900 19342 raft_consensus.cc:686] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 1 LEADER]: Becoming Leader. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Running, Role: LEADER
I20260928 12:58:48.534953 19342 consensus_queue.cc:237] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.535560 18821 catalog_manager.cc:5495] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 469f316555e04010bdb315eb95cf0c0b (127.11.11.67). New cstate: current_term: 1 leader_uuid: "469f316555e04010bdb315eb95cf0c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } health_report { overall_health: HEALTHY } } }
I20260928 12:58:48.584678 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.582331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.11.11.67:34877)
0928 12:58:48.582393 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.582395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.582404 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:48.582457 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.582458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.582467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.582515 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.582646 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.583102 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.583854 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.584073 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.584166 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.584362 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.584395 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.584456 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.584517 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.584546 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:58:48.585629 19120 raft_consensus.cc:1261] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Refusing update from remote peer 469f316555e04010bdb315eb95cf0c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.585728 18979 raft_consensus.cc:1261] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Refusing update from remote peer 469f316555e04010bdb315eb95cf0c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.585908 19390 consensus_queue.cc:1035] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.586021 19342 consensus_queue.cc:1035] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.588857 18821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.588953 18821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.589524 18821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Sending DeleteTablet for 3 replicas of tablet a249a5ed779e442baa85a8960af003d7
I20260928 12:58:48.589802 19240 tablet_service.cc:1514] Processing DeleteTablet for tablet a249a5ed779e442baa85a8960af003d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34884
I20260928 12:58:48.589852 18960 tablet_service.cc:1514] Processing DeleteTablet for tablet a249a5ed779e442baa85a8960af003d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:33808
I20260928 12:58:48.589920 19357 tablet_replica.cc:330] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: stopping tablet replica
I20260928 12:58:48.589974 19357 raft_consensus.cc:2229] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:48.589967 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet a249a5ed779e442baa85a8960af003d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53476
I20260928 12:58:48.590046 19357 raft_consensus.cc:2258] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.590070 19359 tablet_replica.cc:330] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: stopping tablet replica
I20260928 12:58:48.590133 19359 raft_consensus.cc:2229] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.590189 19359 raft_consensus.cc:2258] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.590090 18821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:58:48.590261 18821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:58:48.590276 19358 tablet_replica.cc:330] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: stopping tablet replica
I20260928 12:58:48.590337 19358 raft_consensus.cc:2229] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.590389 19358 raft_consensus.cc:2258] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.590477 19357 ts_tablet_manager.cc:1905] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.590835 19359 ts_tablet_manager.cc:1905] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.590895 19358 ts_tablet_manager.cc:1905] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.591455 19240 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdf270385d2455ca788d181ea3c1811 (DEFAULT_TABLE table=test-table [id=f6133cf4a9084ce393fe54a369d3a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.591553 19240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdf270385d2455ca788d181ea3c1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.592165 19357 ts_tablet_manager.cc:1918] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:48.592237 19357 log.cc:1196] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/a249a5ed779e442baa85a8960af003d7
I20260928 12:58:48.592227 18960 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdf270385d2455ca788d181ea3c1811 (DEFAULT_TABLE table=test-table [id=f6133cf4a9084ce393fe54a369d3a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.592324 18960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdf270385d2455ca788d181ea3c1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.592473 19357 ts_tablet_manager.cc:1939] T a249a5ed779e442baa85a8960af003d7 P 469f316555e04010bdb315eb95cf0c0b: Deleting consensus metadata
I20260928 12:58:48.592470 19359 ts_tablet_manager.cc:1918] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:48.592470 19358 ts_tablet_manager.cc:1918] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:58:48.592530 19358 log.cc:1196] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/a249a5ed779e442baa85a8960af003d7
I20260928 12:58:48.592538 19359 log.cc:1196] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/a249a5ed779e442baa85a8960af003d7
I20260928 12:58:48.592770 19358 ts_tablet_manager.cc:1939] T a249a5ed779e442baa85a8960af003d7 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting consensus metadata
I20260928 12:58:48.592770 19359 ts_tablet_manager.cc:1939] T a249a5ed779e442baa85a8960af003d7 P 27f3496526ce4233a7c0f4070e3b5b6e: Deleting consensus metadata
I20260928 12:58:48.593518 19334 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: Bootstrap starting.
I20260928 12:58:48.593623 18807 catalog_manager.cc:4841] TS 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877): tablet a249a5ed779e442baa85a8960af003d7 (table test-table [id=371eb9e033aa4a6dbea1179c0da31d36]) successfully deleted
I20260928 12:58:48.593760 18808 catalog_manager.cc:4841] TS 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873): tablet a249a5ed779e442baa85a8960af003d7 (table test-table [id=371eb9e033aa4a6dbea1179c0da31d36]) successfully deleted
I20260928 12:58:48.593879 19334 tablet_bootstrap.cc:654] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.593899 19100 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdf270385d2455ca788d181ea3c1811 (DEFAULT_TABLE table=test-table [id=f6133cf4a9084ce393fe54a369d3a77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:58:48.594014 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdf270385d2455ca788d181ea3c1811. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:48.594447 19334 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: No bootstrap required, opened a new log
I20260928 12:58:48.594498 19334 ts_tablet_manager.cc:1397] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.594558 19338 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Bootstrap starting.
I20260928 12:58:48.594615 19334 raft_consensus.cc:348] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.594691 19334 raft_consensus.cc:374] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.594712 19334 raft_consensus.cc:729] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.594759 19334 consensus_queue.cc:260] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.594863 19334 ts_tablet_manager.cc:1428] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:48.594905 19338 tablet_bootstrap.cc:654] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.595270 19336 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Bootstrap starting.
I20260928 12:58:48.595407 18810 catalog_manager.cc:4841] TS 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101): tablet a249a5ed779e442baa85a8960af003d7 (table test-table [id=371eb9e033aa4a6dbea1179c0da31d36]) successfully deleted
I20260928 12:58:48.595584 19336 tablet_bootstrap.cc:654] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:48.595875 19338 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: No bootstrap required, opened a new log
I20260928 12:58:48.595950 19338 ts_tablet_manager.cc:1397] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.596089 19338 raft_consensus.cc:348] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.596172 19338 raft_consensus.cc:374] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.596186 19336 tablet_bootstrap.cc:492] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: No bootstrap required, opened a new log
I20260928 12:58:48.596212 19338 raft_consensus.cc:729] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469f316555e04010bdb315eb95cf0c0b, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.596257 19336 ts_tablet_manager.cc:1397] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:48.596267 19338 consensus_queue.cc:260] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.596369 19338 ts_tablet_manager.cc:1428] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:48.596370 19336 raft_consensus.cc:348] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.596446 19336 raft_consensus.cc:374] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:48.596470 19336 raft_consensus.cc:729] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e7270ac344d01bc12e1ee9e0c79d7, State: Initialized, Role: FOLLOWER
I20260928 12:58:48.596515 19336 consensus_queue.cc:260] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.596621 19336 ts_tablet_manager.cc:1428] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:48.852589 19339 raft_consensus.cc:482] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:58:48.852720 19339 raft_consensus.cc:504] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.852949 19339 leader_election.cc:288] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873), 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877)
I20260928 12:58:48.853195 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdf270385d2455ca788d181ea3c1811" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b" is_pre_election: true
I20260928 12:58:48.853277 19260 raft_consensus.cc:2454] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 0.
I20260928 12:58:48.853200 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdf270385d2455ca788d181ea3c1811" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" is_pre_election: true
I20260928 12:58:48.853330 19120 raft_consensus.cc:2454] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 0.
I20260928 12:58:48.853466 18911 leader_election.cc:302] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:48.853581 19339 raft_consensus.cc:2781] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:58:48.853669 19339 raft_consensus.cc:482] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:58:48.853701 19339 raft_consensus.cc:3037] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.854396 19339 raft_consensus.cc:504] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.854550 19339 leader_election.cc:288] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 election: Requested vote from peers 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873), 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877)
I20260928 12:58:48.854733 19120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdf270385d2455ca788d181ea3c1811" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7"
I20260928 12:58:48.854737 19260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdf270385d2455ca788d181ea3c1811" candidate_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469f316555e04010bdb315eb95cf0c0b"
I20260928 12:58:48.854799 19260 raft_consensus.cc:3037] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.854815 19120 raft_consensus.cc:3037] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:58:48.855400 19120 raft_consensus.cc:2454] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 1.
I20260928 12:58:48.855427 19260 raft_consensus.cc:2454] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3496526ce4233a7c0f4070e3b5b6e in term 1.
I20260928 12:58:48.855599 18911 leader_election.cc:302] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3496526ce4233a7c0f4070e3b5b6e, 469f316555e04010bdb315eb95cf0c0b; no voters:
I20260928 12:58:48.855719 19339 raft_consensus.cc:2781] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:48.855829 19339 raft_consensus.cc:686] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 LEADER]: Becoming Leader. State: Replica: 27f3496526ce4233a7c0f4070e3b5b6e, State: Running, Role: LEADER
I20260928 12:58:48.855901 19339 consensus_queue.cc:237] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } }
I20260928 12:58:48.856439 18821 catalog_manager.cc:5495] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65). New cstate: current_term: 1 leader_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f3496526ce4233a7c0f4070e3b5b6e" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20260928 12:58:48.882951 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.880920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.11.11.65:39101)
0928 12:58:48.881017 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.881019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.881027 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:48.881055 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.881056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.881066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.881119 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.881226 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.881701 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.882384 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.882609 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.882662 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.882762 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.882791 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.882826 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.882868 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.882884 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260928 12:58:48.887599 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.885229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.11.11.65:47821)
0928 12:58:48.885378 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.885388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:48.885461 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:48.885577 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:48.885577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:48.885583 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:58:48.885625 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.885626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.886270 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.886271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.886826 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:48.886828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.887205 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.887206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:48.887257 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.887261 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.887414 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.887415 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:58:48.887461 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:58:48.887516 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":5,"thread_start_us":70,"threads_started":1}
I20260928 12:58:48.887599 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.885248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47821 (local address 127.11.11.65:39101)
0928 12:58:48.885326 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:48.885329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:48.885448 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:58:48.885523 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:48.885523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:48.885533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:48.885577 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:58:48.885672 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.886193 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.886908 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:48.887131 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:48.887204 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:48.887335 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:58:48.887366 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:58:48.887414 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:48.887463 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 12:58:48.887505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":38}
I20260928 12:58:48.889794 19120 raft_consensus.cc:1261] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Refusing update from remote peer 27f3496526ce4233a7c0f4070e3b5b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.889789 19260 raft_consensus.cc:1261] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Refusing update from remote peer 27f3496526ce4233a7c0f4070e3b5b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:58:48.890015 19339 consensus_queue.cc:1035] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "469f316555e04010bdb315eb95cf0c0b" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.890107 19418 consensus_queue.cc:1035] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2e7270ac344d01bc12e1ee9e0c79d7" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:58:48.901108 18821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.901217 18821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:58:48.901854 18821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9907617e03dd4daa91ede01d921a0b66: Sending DeleteTablet for 3 replicas of tablet 8fdf270385d2455ca788d181ea3c1811
I20260928 12:58:48.902098 18960 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdf270385d2455ca788d181ea3c1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:33808
I20260928 12:58:48.902158 19100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdf270385d2455ca788d181ea3c1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53476
I20260928 12:58:48.902206 19359 tablet_replica.cc:330] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e: stopping tablet replica
I20260928 12:58:48.902164 19240 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdf270385d2455ca788d181ea3c1811 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:58:48 UTC) from {username='oryx', principal='oryx/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34884
I20260928 12:58:48.902257 19359 raft_consensus.cc:2229] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:58:48.902275 19357 tablet_replica.cc:330] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: stopping tablet replica
I20260928 12:58:48.902323 19357 raft_consensus.cc:2229] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.902341 19359 raft_consensus.cc:2258] T 8fdf270385d2455ca788d181ea3c1811 P 27f3496526ce4233a7c0f4070e3b5b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.902359 19357 raft_consensus.cc:2258] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.902422 19358 tablet_replica.cc:330] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: stopping tablet replica
I20260928 12:58:48.902486 19358 raft_consensus.cc:2229] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:58:48.902521 19358 raft_consensus.cc:2258] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:58:48.902853 19357 ts_tablet_manager.cc:1905] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.903105 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18895
I20260928 12:58:48.904670 19357 ts_tablet_manager.cc:1918] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:48.904734 19357 log.cc:1196] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/8fdf270385d2455ca788d181ea3c1811
I20260928 12:58:48.904956 19357 ts_tablet_manager.cc:1939] T 8fdf270385d2455ca788d181ea3c1811 P 469f316555e04010bdb315eb95cf0c0b: Deleting consensus metadata
I20260928 12:58:48.905318 19358 ts_tablet_manager.cc:1905] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:58:48.906463 19358 ts_tablet_manager.cc:1918] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:58:48.906540 19358 log.cc:1196] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/8fdf270385d2455ca788d181ea3c1811
I20260928 12:58:48.906759 19358 ts_tablet_manager.cc:1939] T 8fdf270385d2455ca788d181ea3c1811 P 4f2e7270ac344d01bc12e1ee9e0c79d7: Deleting consensus metadata
I20260928 12:58:48.907096 18807 catalog_manager.cc:4841] TS 469f316555e04010bdb315eb95cf0c0b (127.11.11.67:34877): tablet 8fdf270385d2455ca788d181ea3c1811 (table test-table [id=f6133cf4a9084ce393fe54a369d3a77a]) successfully deleted
I20260928 12:58:48.907096 18808 catalog_manager.cc:4841] TS 4f2e7270ac344d01bc12e1ee9e0c79d7 (127.11.11.66:41873): tablet 8fdf270385d2455ca788d181ea3c1811 (table test-table [id=f6133cf4a9084ce393fe54a369d3a77a]) successfully deleted
W20260928 12:58:48.910508 18810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:39101 (error 108)
I20260928 12:58:48.910807 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19035
I20260928 12:58:48.911494 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:48.910931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:39101 (local address 127.0.0.1:33838)
0928 12:58:48.911396 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:48.911442 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:39101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":40,"threads_started":1}
W20260928 12:58:48.912317 18810 catalog_manager.cc:4584] TS 27f3496526ce4233a7c0f4070e3b5b6e (127.11.11.65:39101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fdf270385d2455ca788d181ea3c1811: Network error: Client connection negotiation failed: client connection to 127.11.11.65:39101: connect: Connection refused (error 111)
I20260928 12:58:48.917032 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19175
I20260928 12:58:48.922714 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 18791
2026-09-28T12:58:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3115 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[19433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[19433](info): set up 2 sockets
Sep 28 12:58:48 dist-test-slave-8sjz krb5kdc[19433](info): commencing operation
krb5kdc: starting...
W20260928 12:58:49.993988 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:50 dist-test-slave-8sjz krb5kdc[19433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:50.117229 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46879
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:50.199826 19446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:50.200112 19446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:50.200160 19446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:50.201366 19446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:50.201414 19446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:50.201429 19446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:50.201442 19446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:50.201503 19446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:50.203130 19446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:50.203398 19446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:50.203671 19446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:58:50.208705 19446 server_base.cc:1034] running on GCE node
W20260928 12:58:50.208678 19451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:50.208668 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:50.208674 19452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:50.209188 19446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:50.209556 19446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:50.210727 19446 hybrid_clock.cc:648] HybridClock initialized: now 1790600330210709 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T12:58:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1290 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:58:50 dist-test-slave-8sjz krb5kdc[19459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:58:50 dist-test-slave-8sjz krb5kdc[19459](info): set up 2 sockets
Sep 28 12:58:50 dist-test-slave-8sjz krb5kdc[19459](info): commencing operation
krb5kdc: starting...
W20260928 12:58:51.253808 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:58:51.379168 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43429
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:43429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:51.466141 19472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:51.466423 19472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:51.466465 19472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:51.467705 19472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:51.467790 19472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:51.467818 19472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:51.467849 19472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:51.467875 19472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:51.469532 19472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:43429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:51.469857 19472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:51.470106 19472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:51.475122 19477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.475251 19472 server_base.cc:1034] running on GCE node
W20260928 12:58:51.475144 19478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.475122 19480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.475668 19472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:51.475943 19472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:51.477099 19472 hybrid_clock.cc:648] HybridClock initialized: now 1790600331477085 us; error 47 us; skew 500 ppm
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:51.479605 19472 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:51.480160 19472 webserver.cc:460] Webserver started at http://127.11.11.126:41355/ using document root <none> and password file <none>
I20260928 12:58:51.480370 19472 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:51.480413 19472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:51.480506 19472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:51.481561 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "e310c93b594446b6a9cc5b3b8521827e"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "7e293885858cc4bf963b44b256da05d9"
server_key_iv: "cb6a2fc7a9e12adaaba23334ae91df35"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.481878 19472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "e310c93b594446b6a9cc5b3b8521827e"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "7e293885858cc4bf963b44b256da05d9"
server_key_iv: "cb6a2fc7a9e12adaaba23334ae91df35"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.483191 19472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:51.483973 19487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.484164 19472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:51.484261 19472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "e310c93b594446b6a9cc5b3b8521827e"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "7e293885858cc4bf963b44b256da05d9"
server_key_iv: "cb6a2fc7a9e12adaaba23334ae91df35"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.484323 19472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:51.495929 19472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:51.497892 19472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:51.498010 19472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:51.502014 19472 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:43429
I20260928 12:58:51.502071 19539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:43429 every 8 connection(s)
I20260928 12:58:51.502449 19472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:51.503367 19540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:58:51.504233 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19472
I20260928 12:58:51.504317 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:58:51.504519 11309 external_mini_cluster.cc:1428] Setting key 540312afafa6ee95bc116e987cf02ff3
I20260928 12:58:51.506876 19540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Bootstrap starting.
I20260928 12:58:51.507503 19540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Neither blocks nor log segments found. Creating new log.
I20260928 12:58:51.507815 19540 log.cc:824] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:51.508553 19540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: No bootstrap required, opened a new log
I20260928 12:58:51.510624 19540 raft_consensus.cc:348] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:51.510804 19540 raft_consensus.cc:374] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:58:51.510869 19540 raft_consensus.cc:729] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e310c93b594446b6a9cc5b3b8521827e, State: Initialized, Role: FOLLOWER
I20260928 12:58:51.511039 19540 consensus_queue.cc:260] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:51.511133 19540 raft_consensus.cc:388] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:51.511176 19540 raft_consensus.cc:482] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:51.511226 19540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:51.511884 19540 raft_consensus.cc:504] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:51.511999 19540 leader_election.cc:302] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e310c93b594446b6a9cc5b3b8521827e; no voters:
I20260928 12:58:51.512224 19540 leader_election.cc:288] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:58:51.512297 19545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:58:51.512454 19545 raft_consensus.cc:686] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 LEADER]: Becoming Leader. State: Replica: e310c93b594446b6a9cc5b3b8521827e, State: Running, Role: LEADER
I20260928 12:58:51.512511 19540 sys_catalog.cc:564] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:51.512571 19545 consensus_queue.cc:237] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:51.512898 19545 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310c93b594446b6a9cc5b3b8521827e. Latest consensus state: current_term: 1 leader_uuid: "e310c93b594446b6a9cc5b3b8521827e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } } }
I20260928 12:58:51.512972 19545 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:51.513129 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e310c93b594446b6a9cc5b3b8521827e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } } }
I20260928 12:58:51.513224 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:51.513447 19551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:51.514164 19551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:51.516157 19551 catalog_manager.cc:1347] Generated new cluster ID: e773eaf171de4ac6a6e7e105a5319ccd
I20260928 12:58:51.516213 19551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:51.516959 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:51.505209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.11.11.126:43429)
0928 12:58:51.505388 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:51.505397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:51.505423 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:58:51.505841 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:51.505848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:51.505895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:51.506008 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:51.506824 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.507834 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.508736 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.509173 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.509296 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:51.511916 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:51.511920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:51.511921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:51.511934 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:51.514609 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.515104 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.515107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.515107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.515224 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.515539 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.515539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.515540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.515997 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:51.516561 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:51.516628 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 12:58:51.516767 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 12:58:51.538476 19551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:58:51.539068 19551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:51.542951 19551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Generated new TSK 0
I20260928 12:58:51.543176 19551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:58:51.574929 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:51.676786 19566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:51.677040 19566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:51.677089 19566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:51.678313 19566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:51.678382 19566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:51.678429 19566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:51.679937 19566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:51.680231 19566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:51.680491 19566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:51.685544 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.685537 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.685537 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.685791 19566 server_base.cc:1034] running on GCE node
I20260928 12:58:51.686012 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:51.686268 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:51.687433 19566 hybrid_clock.cc:648] HybridClock initialized: now 1790600331687410 us; error 38 us; skew 500 ppm
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:51.690129 19566 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:58:51.690598 19566 webserver.cc:460] Webserver started at http://127.11.11.65:33199/ using document root <none> and password file <none>
I20260928 12:58:51.690861 19566 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:51.690932 19566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:51.691079 19566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:51.692235 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "3e2848d10b43422d8be47c267caedaba"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "b42813c080fe1a735d5c9b8f910633e1"
server_key_iv: "f404331889312904d311d5006149bb6b"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.692598 19566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2848d10b43422d8be47c267caedaba"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "b42813c080fe1a735d5c9b8f910633e1"
server_key_iv: "f404331889312904d311d5006149bb6b"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.693935 19566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:58:51.694650 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.694818 19566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:58:51.694922 19566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "3e2848d10b43422d8be47c267caedaba"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "b42813c080fe1a735d5c9b8f910633e1"
server_key_iv: "f404331889312904d311d5006149bb6b"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.694996 19566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:51.708582 19566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:51.711102 19566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:51.711280 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:51.711536 19566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:51.711870 19566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:51.711925 19566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.711964 19566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:51.712005 19566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.717934 19566 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:46645
I20260928 12:58:51.717989 19694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:46645 every 8 connection(s)
I20260928 12:58:51.718358 19566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:51.720885 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19566
I20260928 12:58:51.721021 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:58:51.721215 11309 external_mini_cluster.cc:1428] Setting key 9e0239eaaad430597776b1a5bb2c19cb
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:51.730057 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:51.719252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:51231 (local address 127.11.11.126:43429)
0928 12:58:51.719303 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:51.719305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:51.719533 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:58:51.720123 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:51.720124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:51.720145 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:51.720210 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:51.721305 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.721934 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.723584 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.723979 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.724083 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:51.726773 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:51.726775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:51.726776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:51.726786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:51.728580 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.728999 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.729000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.729000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.729060 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.729311 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.729312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.729313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.729468 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:51.729549 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:51.729859 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 12:58:51.729961 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:58:51.730175 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:51.719204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:51231)
0928 12:58:51.719393 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:51.719440 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:51.719966 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:51.720290 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:51.720294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:51.720918 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:51.721196 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:51.721203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.722050 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.722051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:51.723443 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:51.723447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.724121 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.724122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:51.724235 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:51.724896 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:51.724905 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:51.726661 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:51.728673 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:51.728681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:51.728684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:51.728918 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:51.729132 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:51.729133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:51.729133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:51.729241 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:51.729570 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:51.729576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:51.729706 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 12:58:51.729808 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:58:51.729945 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 12:58:51.730759 19695 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43429
I20260928 12:58:51.730875 19695 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:51.731068 19695 heartbeater.cc:511] Master 127.11.11.126:43429 requested a full tablet report, sending...
I20260928 12:58:51.731696 19504 ts_manager.cc:194] Registered new tserver with Master: 3e2848d10b43422d8be47c267caedaba (127.11.11.65:46645)
I20260928 12:58:51.732483 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:51231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:58:51.751075 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:51.833945 19700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:51.834211 19700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:51.834254 19700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:51.835536 19700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:51.835598 19700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:51.835645 19700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:51.837271 19700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:51.837543 19700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:51.837787 19700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:51.842864 19706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.843019 19705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.843019 19708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.843262 19700 server_base.cc:1034] running on GCE node
I20260928 12:58:51.843483 19700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:51.843731 19700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:51.844872 19700 hybrid_clock.cc:648] HybridClock initialized: now 1790600331844860 us; error 26 us; skew 500 ppm
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:51.847422 19700 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:58:51.847827 19700 webserver.cc:460] Webserver started at http://127.11.11.66:41921/ using document root <none> and password file <none>
I20260928 12:58:51.848021 19700 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:51.848121 19700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:51.848224 19700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:51.849290 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "9f4bafedf05b4d59a55e1ec9b6fe274a"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "0f6a875aa13e8bd7f1ec4a5620a38de7"
server_key_iv: "387c3537542078d46a2e7b05456a30fc"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.849596 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "9f4bafedf05b4d59a55e1ec9b6fe274a"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "0f6a875aa13e8bd7f1ec4a5620a38de7"
server_key_iv: "387c3537542078d46a2e7b05456a30fc"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.850912 19700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:51.851668 19715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.851882 19700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:51.851996 19700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "9f4bafedf05b4d59a55e1ec9b6fe274a"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "0f6a875aa13e8bd7f1ec4a5620a38de7"
server_key_iv: "387c3537542078d46a2e7b05456a30fc"
server_key_version: "encryptionkey@0"
I20260928 12:58:51.852136 19700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:51.858922 19700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:51.860837 19700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:51.860998 19700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:51.861238 19700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:51.861554 19700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:51.861608 19700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.861647 19700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:51.861675 19700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:51.867620 19700 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:44395
I20260928 12:58:51.867669 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:44395 every 8 connection(s)
I20260928 12:58:51.867993 19700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:51.876334 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19700
I20260928 12:58:51.876442 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:58:51.876730 11309 external_mini_cluster.cc:1428] Setting key 2540ad708b14a1fddbc6607c0a89a7cd
I20260928 12:58:51.877771 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:51.868819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:34611 (local address 127.11.11.126:43429)
0928 12:58:51.868895 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:51.868898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:51.869033 (+ 135us) server_negotiation.cc:407] Connection header received
0928 12:58:51.869562 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:51.869563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:51.869575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:51.869631 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:51.870622 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.871131 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.872275 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.872475 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.872574 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:51.874778 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:51.874779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:51.874780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:51.874786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:51.876141 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.876652 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.876653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.876653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.876728 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:51.877021 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:51.877022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:51.877022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:51.877149 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:51.877228 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:51.877557 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 12:58:51.877652 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:58:51.877879 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:51.868760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:34611)
0928 12:58:51.868923 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:51.868959 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:51.869457 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:51.869711 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:51.869714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:51.870300 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:51.870506 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:51.870511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.871234 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.871235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:51.872192 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:51.872194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:51.872586 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:51.872587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:51.872688 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:51.873254 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:51.873263 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:51.874694 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:51.876232 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:51.876240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:51.876242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:51.876575 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:51.876801 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:51.876802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:51.876803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:51.876952 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:51.877220 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:51.877227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:51.877398 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 12:58:51.877518 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 12:58:51.877669 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260928 12:58:51.878515 19829 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43429
I20260928 12:58:51.878626 19829 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:51.878806 19829 heartbeater.cc:511] Master 127.11.11.126:43429 requested a full tablet report, sending...
I20260928 12:58:51.879223 19504 ts_manager.cc:194] Registered new tserver with Master: 9f4bafedf05b4d59a55e1ec9b6fe274a (127.11.11.66:44395)
I20260928 12:58:51.879777 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:34611
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:58:51.904223 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:51.988560 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:51.988809 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:51.988852 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:51.990121 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:51.990191 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:51.990238 19834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:51.991760 19834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:51.992085 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:51.992340 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:51.997462 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.997604 19834 server_base.cc:1034] running on GCE node
W20260928 12:58:51.997511 19840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:51.997700 19839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:51.997994 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:51.998289 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:51.999441 19834 hybrid_clock.cc:648] HybridClock initialized: now 1790600331999428 us; error 43 us; skew 500 ppm
Sep 28 12:58:51 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:52.002907 19834 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:58:52.003557 19834 webserver.cc:460] Webserver started at http://127.11.11.67:32777/ using document root <none> and password file <none>
I20260928 12:58:52.003855 19834 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:52.003930 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:52.004113 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:58:52.005244 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "66c8e6d4cac74a309cea0db90634773f"
format_stamp: "Formatted at 2026-09-28 12:58:52 on dist-test-slave-8sjz"
server_key: "8fe4083c85feda0733f642ac9f87f8f9"
server_key_iv: "b9a2a9592b454899e4f2198ee177ce02"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.005620 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "66c8e6d4cac74a309cea0db90634773f"
format_stamp: "Formatted at 2026-09-28 12:58:52 on dist-test-slave-8sjz"
server_key: "8fe4083c85feda0733f642ac9f87f8f9"
server_key_iv: "b9a2a9592b454899e4f2198ee177ce02"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.007025 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:58:52.008160 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.008337 19834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:52.008430 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "66c8e6d4cac74a309cea0db90634773f"
format_stamp: "Formatted at 2026-09-28 12:58:52 on dist-test-slave-8sjz"
server_key: "8fe4083c85feda0733f642ac9f87f8f9"
server_key_iv: "b9a2a9592b454899e4f2198ee177ce02"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.008507 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:52.034255 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:52.036669 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:52.036842 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:52.037089 19834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:52.037441 19834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:52.037482 19834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.037506 19834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:52.037554 19834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.043738 19834 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35507
I20260928 12:58:52.044237 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:52.044226 19962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35507 every 8 connection(s)
I20260928 12:58:52.049436 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19834
I20260928 12:58:52.049515 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:58:52.049705 11309 external_mini_cluster.cc:1428] Setting key a5ce2216afd4f02d19dc6886b5add2d3
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600331, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:58:52.053805 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.045091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:58587 (local address 127.11.11.126:43429)
0928 12:58:52.045166 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.045168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.045389 (+ 221us) server_negotiation.cc:407] Connection header received
0928 12:58:52.045876 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.045877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.045887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.045954 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.046893 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.047389 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.048460 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.048664 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.048721 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.051251 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:52.051253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:52.051253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:52.051260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:52.052623 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:52.053001 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:52.053002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:52.053002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:52.053056 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:52.053270 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:52.053270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:52.053270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:52.053375 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:52.053425 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:52.053655 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 12:58:52.053740 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:58:52.054046 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.045083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:58587)
0928 12:58:52.045259 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.045303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.045791 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.046013 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.046016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.046604 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.046810 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.046815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.047473 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.047473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.048385 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.048387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.048739 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.048739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.048842 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.049351 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.049359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.051142 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:58:52.052704 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:52.052710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:52.052712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:52.052933 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:52.053112 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:58:52.053113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:58:52.053113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:58:52.053215 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:58:52.053428 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:58:52.053432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:58:52.053550 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 12:58:52.053652 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:58:52.053776 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 12:58:52.054592 19963 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43429
I20260928 12:58:52.054702 19963 heartbeater.cc:464] Registering TS with master...
I20260928 12:58:52.054916 19963 heartbeater.cc:511] Master 127.11.11.126:43429 requested a full tablet report, sending...
I20260928 12:58:52.055320 19504 ts_manager.cc:194] Registered new tserver with Master: 66c8e6d4cac74a309cea0db90634773f (127.11.11.67:35507)
I20260928 12:58:52.055712 19504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:58587
I20260928 12:58:52.061697 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
Entry for principal oryx/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab.
WARNING: no policy specified for oryx/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
Entry for principal oryx/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab.
WARNING: no policy specified for oryx/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
Entry for principal oryx/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab.
I20260928 12:58:52.140945 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19566
I20260928 12:58:52.145748 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19700
I20260928 12:58:52.150189 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19834
I20260928 12:58:52.154552 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19472
I20260928 12:58:52.160210 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43429
--webserver_interface=127.11.11.126
--webserver_port=41355
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:43429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:58:52.243213 19972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:52.243525 19972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:52.243568 19972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:52.244904 19972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:58:52.244994 19972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:52.245020 19972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:52.245044 19972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:58:52.245072 19972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:58:52.246796 19972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:43429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=41355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:52.247129 19972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:52.247385 19972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:52.252462 19980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.252478 19977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.252460 19978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:52.252645 19972 server_base.cc:1034] running on GCE node
I20260928 12:58:52.252914 19972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:52.253188 19972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:52.254334 19972 hybrid_clock.cc:648] HybridClock initialized: now 1790600332254312 us; error 43 us; skew 500 ppm
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600332, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:52.263079 19972 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:58:52.263635 19972 webserver.cc:460] Webserver started at http://127.11.11.126:41355/ using document root <none> and password file <none>
I20260928 12:58:52.263844 19972 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:52.263901 19972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:52.265393 19972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:52.266103 19987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.266315 19972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:52.266412 19972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "e310c93b594446b6a9cc5b3b8521827e"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "7e293885858cc4bf963b44b256da05d9"
server_key_iv: "cb6a2fc7a9e12adaaba23334ae91df35"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.266826 19972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:52.275398 19972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:52.277546 19972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:52.277669 19972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:52.281620 19972 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:43429
I20260928 12:58:52.281685 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:43429 every 8 connection(s)
I20260928 12:58:52.282063 19972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:58:52.283687 20040 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:58:52.284626 20040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Bootstrap starting.
I20260928 12:58:52.285424 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 19972
I20260928 12:58:52.285734 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:46645
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:43429
--webserver_port=33199
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:58:52.293633 20040 log.cc:824] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:58:52.295394 20040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:58:52.295706 20040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Bootstrap complete.
I20260928 12:58:52.298072 20040 raft_consensus.cc:348] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:52.298301 20040 raft_consensus.cc:729] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e310c93b594446b6a9cc5b3b8521827e, State: Initialized, Role: FOLLOWER
I20260928 12:58:52.298475 20040 consensus_queue.cc:260] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:52.298545 20040 raft_consensus.cc:388] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:58:52.298617 20040 raft_consensus.cc:482] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:58:52.298664 20040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:58:52.299763 20040 raft_consensus.cc:504] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:52.299853 20040 leader_election.cc:302] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e310c93b594446b6a9cc5b3b8521827e; no voters:
I20260928 12:58:52.300083 20040 leader_election.cc:288] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:58:52.300127 20044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:58:52.300261 20044 raft_consensus.cc:686] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [term 2 LEADER]: Becoming Leader. State: Replica: e310c93b594446b6a9cc5b3b8521827e, State: Running, Role: LEADER
I20260928 12:58:52.300366 20040 sys_catalog.cc:564] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:58:52.300349 20044 consensus_queue.cc:237] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } }
I20260928 12:58:52.300639 20046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e310c93b594446b6a9cc5b3b8521827e. Latest consensus state: current_term: 2 leader_uuid: "e310c93b594446b6a9cc5b3b8521827e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } } }
I20260928 12:58:52.300870 20046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:52.300786 20045 sys_catalog.cc:455] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e310c93b594446b6a9cc5b3b8521827e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e310c93b594446b6a9cc5b3b8521827e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43429 } } }
I20260928 12:58:52.301229 20045 sys_catalog.cc:458] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e [sys.catalog]: This master's current role is: LEADER
I20260928 12:58:52.302521 20060 catalog_manager.cc:1259] Loaded cluster ID: e773eaf171de4ac6a6e7e105a5319ccd
I20260928 12:58:52.302583 20060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: loading cluster ID for follower catalog manager: success
I20260928 12:58:52.303382 20060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: acquiring CA information for follower catalog manager: success
I20260928 12:58:52.303705 20060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:58:52.303795 20050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:58:52.303936 20050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:58:52.304026 20050 catalog_manager.cc:1259] Loaded cluster ID: e773eaf171de4ac6a6e7e105a5319ccd
I20260928 12:58:52.304080 20050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:58:52.304246 20050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:58:52.304352 20050 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e310c93b594446b6a9cc5b3b8521827e: Loaded TSK: 0
I20260928 12:58:52.304512 20050 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:58:52.397109 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:52.397325 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:52.397352 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:52.398602 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:52.398651 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:52.398679 20042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:58:52.400178 20042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.65.keytab
--principal=oryx/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:46645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=33199
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:52.400472 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:52.400736 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:52.406019 20066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.406013 20067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.406116 20069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:52.406450 20042 server_base.cc:1034] running on GCE node
I20260928 12:58:52.406672 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:52.406976 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:52.408123 20042 hybrid_clock.cc:648] HybridClock initialized: now 1790600332408102 us; error 35 us; skew 500 ppm
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600332, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:52.411466 20042 init.cc:377] Logged in from keytab as oryx/127.11.11.65@KRBTEST.COM (short username oryx)
I20260928 12:58:52.412114 20042 webserver.cc:460] Webserver started at http://127.11.11.65:33199/ using document root <none> and password file <none>
I20260928 12:58:52.412421 20042 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:52.412513 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:52.414400 20042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:58:52.415162 20076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.415387 20042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:58:52.415490 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "3e2848d10b43422d8be47c267caedaba"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "b42813c080fe1a735d5c9b8f910633e1"
server_key_iv: "f404331889312904d311d5006149bb6b"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.415944 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:52.439255 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:52.441920 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:52.442082 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:52.442369 20042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:52.442754 20042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:52.442803 20042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.442835 20042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:52.442864 20042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.450176 20042 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:46645
I20260928 12:58:52.450213 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:46645 every 8 connection(s)
I20260928 12:58:52.450654 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:58:52.451766 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20042
I20260928 12:58:52.452113 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:44395
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:43429
--webserver_port=41921
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.462601 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.451515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:59289)
0928 12:58:52.452171 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.452222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.452851 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.453389 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.453394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.454178 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.454459 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.454468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.456315 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.456316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.457580 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.457583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.458552 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.458553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.458697 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.459387 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.459398 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.462387 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":86,"threads_started":1}
W20260928 12:58:52.462848 20190 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:43429 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.11.126:43429: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
W20260928 12:58:52.463095 20193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.451598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59289 (local address 127.11.11.126:43429)
0928 12:58:52.452430 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.452441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.452491 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:58:52.453101 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.453109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.453170 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.453322 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.454614 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.456188 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.457686 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.458439 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.458567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.462574 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:59289: BlockingRecv error: recv got EOF from 127.11.11.65:59289 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":95,"threads_started":1}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.468817 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.463098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:46197)
0928 12:58:52.463290 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.463304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.463359 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.463553 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.463554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.463707 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.463763 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.463764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.464726 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.464727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.465571 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.465574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.466055 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.466056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.466120 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.466629 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.466634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.468750 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172}
W20260928 12:58:52.469568 20193 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.463202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:46197 (local address 127.11.11.126:43429)
0928 12:58:52.463242 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.463245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.463416 (+ 171us) server_negotiation.cc:407] Connection header received
0928 12:58:52.463443 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.463444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.463456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.463531 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.463856 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.464625 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.465665 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.465987 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.466059 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.469492 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:46197: BlockingRecv error: recv got EOF from 127.11.11.65:46197 (error 108)
Metrics: {"server-negotiator.queue_time_us":22,"thread_start_us":99,"threads_started":1}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.474568 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.469127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:53807)
0928 12:58:52.469184 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.469196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.469242 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.469480 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.469480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.469619 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.469652 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.469653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.470622 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.470622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.471482 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.471485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.471979 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.471980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.472052 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.472528 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.472533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.474488 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 12:58:52.474699 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.469219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:53807 (local address 127.11.11.126:43429)
0928 12:58:52.469352 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.469354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.469361 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:58:52.469386 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.469386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.469397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.469467 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.469739 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.470538 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.471576 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.471901 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.471971 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.474627 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:53807: BlockingRecv error: recv got EOF from 127.11.11.65:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260928 12:58:52.474953 20190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:58:52.541464 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:52.541723 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:52.541761 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:52.542979 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:52.543046 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:52.543093 20194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:58:52.544606 20194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.66.keytab
--principal=oryx/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:44395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=41921
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:52.544891 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:52.545130 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:52.550310 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.550320 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:52.550449 20194 server_base.cc:1034] running on GCE node
W20260928 12:58:52.550536 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:52.550997 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:52.551280 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:52.552454 20194 hybrid_clock.cc:648] HybridClock initialized: now 1790600332552442 us; error 32 us; skew 500 ppm
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600332, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:52.555106 20194 init.cc:377] Logged in from keytab as oryx/127.11.11.66@KRBTEST.COM (short username oryx)
I20260928 12:58:52.555580 20194 webserver.cc:460] Webserver started at http://127.11.11.66:41921/ using document root <none> and password file <none>
I20260928 12:58:52.555835 20194 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:52.555922 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:52.557461 20194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:52.558173 20210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.558431 20194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:58:52.558583 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "9f4bafedf05b4d59a55e1ec9b6fe274a"
format_stamp: "Formatted at 2026-09-28 12:58:51 on dist-test-slave-8sjz"
server_key: "0f6a875aa13e8bd7f1ec4a5620a38de7"
server_key_iv: "387c3537542078d46a2e7b05456a30fc"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.558970 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:52.568181 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:52.570322 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:52.570485 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:52.570747 20194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:52.571071 20194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:52.571127 20194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.571173 20194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:52.571197 20194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.577425 20194 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:44395
I20260928 12:58:52.577466 20323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:44395 every 8 connection(s)
I20260928 12:58:52.577901 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:58:52.585601 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.578709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:60355 (local address 127.11.11.126:43429)
0928 12:58:52.578755 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.578757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.578983 (+ 226us) server_negotiation.cc:407] Connection header received
0928 12:58:52.579485 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.579486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.579498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.579553 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.580646 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.581158 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.582246 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.582447 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.582507 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.585550 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:60355: BlockingRecv error: recv got EOF from 127.11.11.66:60355 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:58:52.585602 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.578693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:60355)
0928 12:58:52.578852 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.578891 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.579388 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.579612 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.579616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.580307 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.580519 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.580525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.581246 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.581247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.582147 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.582149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.582571 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.582572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.582680 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.583159 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.583167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.585400 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260928 12:58:52.585911 20324 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:43429 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.11.126:43429: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
I20260928 12:58:52.588020 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20194
I20260928 12:58:52.588393 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:35507
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:43429
--webserver_port=32777
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.590706 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.586152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:34969)
0928 12:58:52.586207 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.586219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.586270 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.586407 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.586408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.586543 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.586588 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.586589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.587227 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.587228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.587759 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.587761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.588205 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.588206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.588292 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.588757 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.588762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.590618 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 12:58:52.590760 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.586212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:34969 (local address 127.11.11.126:43429)
0928 12:58:52.586247 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.586249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.586257 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:52.586334 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.586335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.586347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.586403 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.586673 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.587146 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.587861 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.588106 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.588195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.590693 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:34969: BlockingRecv error: recv got EOF from 127.11.11.66:34969 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.595337 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.591027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:40157)
0928 12:58:52.591065 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.591076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.591120 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.591256 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.591257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.591378 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.591405 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.591406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.592145 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.592146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.592642 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.592643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.593015 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.593016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.593068 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.593520 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.593525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.595239 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:58:52.595458 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.591070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:40157 (local address 127.11.11.126:43429)
0928 12:58:52.591127 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.591129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.591139 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:52.591171 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.591172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.591183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.591253 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.591480 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.592018 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.592727 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.592931 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.593020 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.595357 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:40157: BlockingRecv error: recv got EOF from 127.11.11.66:40157 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 12:58:52.595683 20324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:58:52.671001 20327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:58:52.671218 20327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:58:52.671241 20327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:58:52.672487 20327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:58:52.672529 20327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:58:52.672555 20327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:58:52.674007 20327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/krb5kdc/oryx_127.11.11.67.keytab
--principal=oryx/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:35507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=32777
--tserver_master_addrs=127.11.11.126:43429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:58:52.674254 20327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:58:52.674511 20327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:58:52.679613 20332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.679736 20335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:58:52.679857 20333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:58:52.679896 20327 server_base.cc:1034] running on GCE node
I20260928 12:58:52.680167 20327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:58:52.680437 20327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:58:52.681612 20327 hybrid_clock.cc:648] HybridClock initialized: now 1790600332681591 us; error 39 us; skew 500 ppm
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600332, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:58:52.684276 20327 init.cc:377] Logged in from keytab as oryx/127.11.11.67@KRBTEST.COM (short username oryx)
I20260928 12:58:52.684798 20327 webserver.cc:460] Webserver started at http://127.11.11.67:32777/ using document root <none> and password file <none>
I20260928 12:58:52.685072 20327 fs_manager.cc:359] Metadata directory not provided
I20260928 12:58:52.685158 20327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:58:52.686604 20327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:58:52.687292 20342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.687497 20327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:58:52.687604 20327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "66c8e6d4cac74a309cea0db90634773f"
format_stamp: "Formatted at 2026-09-28 12:58:52 on dist-test-slave-8sjz"
server_key: "8fe4083c85feda0733f642ac9f87f8f9"
server_key_iv: "b9a2a9592b454899e4f2198ee177ce02"
server_key_version: "encryptionkey@0"
I20260928 12:58:52.687989 20327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:58:52.713661 20327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:58:52.715736 20327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:58:52.715920 20327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:58:52.716228 20327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:58:52.716544 20327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:58:52.716598 20327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.716672 20327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:58:52.716704 20327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:58:52.722594 20327 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35507
I20260928 12:58:52.722642 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35507 every 8 connection(s)
I20260928 12:58:52.723004 20327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:58:52.723933 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20327
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:58:52.731824 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.723760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:35739 (local address 127.11.11.126:43429)
0928 12:58:52.723848 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.723851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.724110 (+ 259us) server_negotiation.cc:407] Connection header received
0928 12:58:52.724843 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.724844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.724856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.724922 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.726063 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.726783 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.728153 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.728355 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.728431 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.731764 (+ 3333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:35739: BlockingRecv error: recv got EOF from 127.11.11.67:35739 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 12:58:52.731910 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.723719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:35739)
0928 12:58:52.723935 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.723989 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.724732 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.724979 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.724982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.725695 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.725963 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.725969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.726903 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.726904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.727995 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.727998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.728442 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.728443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.728567 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.729049 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.729058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.731690 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
W20260928 12:58:52.732154 20456 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:43429 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.11.126:43429: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
I20260928 12:58:52.732347 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.724493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.11.11.126:43429)
0928 12:58:52.724556 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.724558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.724567 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:52.724623 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.724624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.724638 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.724702 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.725368 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.726127 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.727231 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.727513 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.727584 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.728886 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:58:52.728890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:58:52.728890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:58:52.728921 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:58:52.730612 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:52.731025 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:52.731028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:52.731029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:52.731122 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:58:52.731392 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:58:52.731397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:58:52.731397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:58:52.731625 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:58:52.731809 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 12:58:52.732109 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 12:58:52.732234 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.736959 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.732392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:53485)
0928 12:58:52.732535 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.732546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.732598 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.732718 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.732718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.732863 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.732923 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.732925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.733646 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.733647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.734175 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.734176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.734550 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.734551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.734621 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.735080 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.735085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.736906 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260928 12:58:52.737075 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.732455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:53485 (local address 127.11.11.126:43429)
0928 12:58:52.732491 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.732493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.732601 (+ 108us) server_negotiation.cc:407] Connection header received
0928 12:58:52.732646 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.732647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.732657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.732714 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.733010 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.733573 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.734283 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.734471 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.734550 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.736997 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:53485: BlockingRecv error: recv got EOF from 127.11.11.67:53485 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:52 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:52.741297 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:52.737248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:49775)
0928 12:58:52.737300 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:52.737312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:52.737377 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:52.737506 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:52.737507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:52.737649 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:52.737679 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.737681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.738326 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.738327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.738858 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:52.738860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.739225 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.739226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:52.739278 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.739680 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:52.739684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:52.741210 (+ 1526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:58:52.741346 20195 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:52.737329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:49775 (local address 127.11.11.126:43429)
0928 12:58:52.737380 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:52.737382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:52.737391 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:58:52.737437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:52.737438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:52.737446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:52.737501 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:52.737757 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.738235 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.738940 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:52.739153 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:52.739228 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:52.741281 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:49775: BlockingRecv error: recv got EOF from 127.11.11.67:49775 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 12:58:52.741532 20456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:53.479933 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:53.475382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:43277)
0928 12:58:53.475527 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:53.475539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:53.475603 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:53.475758 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:53.475759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:53.475918 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:53.475970 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.475971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.476705 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.476706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.477284 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.477286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.477616 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.477617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.477659 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.478084 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:53.478089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:53.479857 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
W20260928 12:58:53.480113 20461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:53.475454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:43277 (local address 127.11.11.126:43429)
0928 12:58:53.475611 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:53.475613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:53.475645 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:58:53.475666 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:53.475667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:53.475676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:53.475738 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:53.476072 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.476606 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.477359 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.477551 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.477603 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.479983 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:43277: BlockingRecv error: recv got EOF from 127.11.11.65:43277 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:53.601006 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:53.596138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:57657)
0928 12:58:53.596286 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:53.596304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:53.596370 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:53.596514 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:53.596514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:53.596691 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:53.596744 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.596746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.597382 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.597383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.597967 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.597969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.598330 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.598330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.598373 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.598939 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:53.598943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:53.600936 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260928 12:58:53.601140 20461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:53.596226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:57657 (local address 127.11.11.126:43429)
0928 12:58:53.596289 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:53.596291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:53.596363 (+ 72us) server_negotiation.cc:407] Connection header received
0928 12:58:53.596428 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:53.596429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:53.596441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:53.596495 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:53.596816 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.597306 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.598058 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.598259 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.598317 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.601038 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:57657: BlockingRecv error: recv got EOF from 127.11.11.66:57657 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:53 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:53.746551 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:53.742068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:55959)
0928 12:58:53.742224 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:53.742237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:53.742305 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:53.742451 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:53.742452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:53.742616 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:53.742673 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.742674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.743314 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.743315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.743917 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:53.743919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.744272 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.744273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:53.744314 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.744748 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:53.744752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:53.746477 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260928 12:58:53.746652 20461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:53.742159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:55959 (local address 127.11.11.126:43429)
0928 12:58:53.742238 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:53.742240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:53.742307 (+ 67us) server_negotiation.cc:407] Connection header received
0928 12:58:53.742371 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:53.742372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:53.742386 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:53.742433 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:53.742746 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.743241 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.744000 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:53.744205 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:53.744262 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:53.746580 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:55959: BlockingRecv error: recv got EOF from 127.11.11.67:55959 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:54.485478 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:54.480530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:42391)
0928 12:58:54.480684 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:54.480694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:54.480749 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:54.480986 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:54.480987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:54.481148 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:54.481200 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.481201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.481841 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.481842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.482375 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.482376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.482689 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.482690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.482730 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.483215 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:54.483219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:54.485378 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
W20260928 12:58:54.485541 20465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:54.480656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:42391 (local address 127.11.11.126:43429)
0928 12:58:54.480815 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:54.480817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:54.480834 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:58:54.480861 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:54.480862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:54.480872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:54.480945 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:54.481276 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.481765 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.482448 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.482630 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.482678 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.485466 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:42391: BlockingRecv error: recv got EOF from 127.11.11.65:42391 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:54.606523 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:54.601487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:54219)
0928 12:58:54.601618 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:54.601631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:54.601702 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:54.601909 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:54.601910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:54.602087 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:54.602148 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.602150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.602792 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.602793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.603338 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.603339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.603808 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.603809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.603879 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.604464 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:54.604470 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:54.606461 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260928 12:58:54.606655 20465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:54.601565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:54219 (local address 127.11.11.126:43429)
0928 12:58:54.601616 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:54.601618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:54.601737 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:58:54.601764 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:54.601765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:54.601777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:54.601902 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:54.602222 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.602718 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.603445 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.603718 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.603804 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.606570 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:54219: BlockingRecv error: recv got EOF from 127.11.11.66:54219 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:54 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:54.751708 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:54.747116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:49801)
0928 12:58:54.747268 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:54.747280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:54.747352 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:54.747480 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:54.747481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:54.747670 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:54.747719 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.747720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.748417 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.748418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.748974 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:54.748977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.749317 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.749318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:54.749356 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.749875 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:54.749880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:54.751639 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
W20260928 12:58:54.751847 20465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:54.747215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:49801 (local address 127.11.11.126:43429)
0928 12:58:54.747291 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:54.747293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:54.747351 (+ 58us) server_negotiation.cc:407] Connection header received
0928 12:58:54.747396 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:54.747397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:54.747409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:54.747458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:54.747785 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.748344 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.749053 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:54.749252 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:54.749307 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:54.751763 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:49801: BlockingRecv error: recv got EOF from 127.11.11.67:49801 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:55.490983 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:55.486083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:59521)
0928 12:58:55.486270 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:55.486282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:55.486347 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:55.486534 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:55.486535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:55.486750 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:55.486806 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.486808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.487501 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.487502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.488161 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.488164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.488529 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.488530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.488608 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.489062 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:55.489068 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:55.490913 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":120,"threads_started":1}
W20260928 12:58:55.491154 20469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:55.486243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59521 (local address 127.11.11.126:43429)
0928 12:58:55.486367 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:55.486369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:55.486379 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:55.486414 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:55.486414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:55.486425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:55.486488 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:55.486905 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.487421 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.488254 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.488456 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.488516 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.491039 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:59521: BlockingRecv error: recv got EOF from 127.11.11.65:59521 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:55.612486 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:55.607084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:42975)
0928 12:58:55.607277 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:55.607288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:55.607385 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:55.607550 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:55.607551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:55.607761 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:55.607810 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.607812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.608710 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.608712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.609270 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.609271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.609684 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.609685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.609745 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.610281 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:55.610287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:55.612381 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
W20260928 12:58:55.612572 20469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:55.607186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:42975 (local address 127.11.11.126:43429)
0928 12:58:55.607261 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:55.607263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:55.607363 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:58:55.607448 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:55.607449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:55.607461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:55.607540 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:55.607902 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.608613 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.609369 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.609588 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.609683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.612480 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:42975: BlockingRecv error: recv got EOF from 127.11.11.66:42975 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:55 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:55.757669 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:55.752357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:59449)
0928 12:58:55.752535 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:55.752547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:55.752612 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:55.752754 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:55.752755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:55.752915 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:55.752967 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.752968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.753695 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.753696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.754233 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:55.754235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.754817 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.754818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:55.754893 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.755537 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:55.755542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:55.757564 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
W20260928 12:58:55.757779 20469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:55.752422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:59449 (local address 127.11.11.126:43429)
0928 12:58:55.752518 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:55.752520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:55.752595 (+ 75us) server_negotiation.cc:407] Connection header received
0928 12:58:55.752673 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:55.752674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:55.752686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:55.752731 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:55.753074 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.753580 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.754332 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:55.754690 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:55.754768 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:55.757710 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:59449: BlockingRecv error: recv got EOF from 127.11.11.67:59449 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:56.496735 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:56.491588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:54591)
0928 12:58:56.491764 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:56.491778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:56.491872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:56.492193 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:56.492195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:56.492424 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:56.492488 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.492490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.493245 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.493246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.493825 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.493827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.494166 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.494167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.494209 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.494669 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:56.494674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:56.496673 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
W20260928 12:58:56.496891 20473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:56.491589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:54591 (local address 127.11.11.126:43429)
0928 12:58:56.491767 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:56.491770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:56.491910 (+ 140us) server_negotiation.cc:407] Connection header received
0928 12:58:56.491956 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:56.491957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:56.491972 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:56.492129 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:56.492565 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.493162 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.493907 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.494102 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.494154 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.496791 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:54591: BlockingRecv error: recv got EOF from 127.11.11.65:54591 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:56.617897 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:56.613143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:40345)
0928 12:58:56.613273 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:56.613287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:56.613381 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:56.613526 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:56.613527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:56.613716 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:56.613768 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.613770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.614446 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.614448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.614992 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.614994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.615393 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.615395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.615462 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.615962 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:56.615966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:56.617797 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
W20260928 12:58:56.617966 20473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:56.613181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:40345 (local address 127.11.11.126:43429)
0928 12:58:56.613252 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:56.613255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:56.613362 (+ 107us) server_negotiation.cc:407] Connection header received
0928 12:58:56.613447 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:56.613448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:56.613462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:56.613509 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:56.613846 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.614351 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.615099 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.615305 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.615391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.617894 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:40345: BlockingRecv error: recv got EOF from 127.11.11.66:40345 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:56 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:56.763223 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:56.758167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:33539)
0928 12:58:56.758443 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:56.758455 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:56.758523 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:56.758705 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:56.758706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:56.758892 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:56.758939 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.758940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.759614 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.759615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.760233 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:56.760235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.760614 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.760615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:56.760670 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.761231 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:56.761236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:56.763117 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":195,"threads_started":1}
W20260928 12:58:56.763280 20473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:56.758362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:33539 (local address 127.11.11.126:43429)
0928 12:58:56.758431 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:56.758434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:56.758503 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:58:56.758586 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:56.758587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:56.758601 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:56.758698 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:56.759019 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.759520 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.760318 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:56.760524 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:56.760613 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:56.763214 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:33539: BlockingRecv error: recv got EOF from 127.11.11.67:33539 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:57.509847 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:57.497303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:54163)
0928 12:58:57.497436 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:57.497449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:57.497531 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:57.497717 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:57.497718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:57.497955 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:57.498035 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.498038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.498693 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.498694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.499228 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.499229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.499673 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.499674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.499735 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.507690 (+ 7955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:57.507696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:57.509750 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260928 12:58:57.509935 20477 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:57.497373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:54163 (local address 127.11.11.126:43429)
0928 12:58:57.497478 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:57.497480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:57.497518 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:58:57.497606 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:57.497607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:57.497622 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:57.497706 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:57.498120 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.498621 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.499331 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.499549 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.499617 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.509860 (+ 10243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:54163: BlockingRecv error: recv got EOF from 127.11.11.65:54163 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:57.624172 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:57.618574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:40489)
0928 12:58:57.618766 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:57.618782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:57.618861 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:57.619078 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:57.619079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:57.619253 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:57.619330 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.619331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.620032 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.620086 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.620647 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.620649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.621077 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.621078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.621142 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.621755 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:57.621761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:57.624021 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
W20260928 12:58:57.624315 20477 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:57.618755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:40489 (local address 127.11.11.126:43429)
0928 12:58:57.618856 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:57.618859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:57.618871 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:58:57.618963 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:57.618964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:57.618981 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:57.619064 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:57.619403 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.619933 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.620777 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.620977 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.621037 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.624228 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:40489: BlockingRecv error: recv got EOF from 127.11.11.66:40489 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:57 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:57.768652 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:57.763842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:42585)
0928 12:58:57.764013 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:57.764026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:57.764134 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:57.764301 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:57.764302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:57.764495 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:57.764544 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.764546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.765217 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.765217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.765748 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:57.765750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.766144 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.766145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:57.766186 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.766727 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:57.766731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:57.768584 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
W20260928 12:58:57.768755 20477 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:57.763912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:42585 (local address 127.11.11.126:43429)
0928 12:58:57.763966 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:57.763968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:57.764110 (+ 142us) server_negotiation.cc:407] Connection header received
0928 12:58:57.764209 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:57.764210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:57.764226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:57.764283 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:57.764612 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.765132 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.765878 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:57.766070 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:57.766132 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:57.768685 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:42585: BlockingRecv error: recv got EOF from 127.11.11.67:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:58.515153 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:58.510354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:54027)
0928 12:58:58.510499 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:58.510514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:58.510564 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:58.510724 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:58.510725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:58.510891 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:58.510941 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.510942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.511569 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.511570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.512140 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.512142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.512493 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.512494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.512537 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.512998 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:58.513003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:58.515028 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260928 12:58:58.515189 20481 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:58.510421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:54027 (local address 127.11.11.126:43429)
0928 12:58:58.510550 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:58.510552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:58.510560 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:58:58.510624 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:58.510625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:58.510635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:58.510697 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:58.511006 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.511499 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.512221 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.512422 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.512478 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.515120 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:54027: BlockingRecv error: recv got EOF from 127.11.11.65:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:58.629673 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:58.624697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:54507)
0928 12:58:58.624881 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:58.624893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:58.624986 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:58.625140 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:58.625141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:58.625330 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:58.625380 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.625382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.626110 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.626111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.626668 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.626670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.627049 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.627050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.627116 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.627654 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:58.627659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:58.629599 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
W20260928 12:58:58.629814 20481 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:58.624793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:54507 (local address 127.11.11.126:43429)
0928 12:58:58.624866 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:58.624868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:58.624967 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:58:58.625048 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:58.625049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:58.625068 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:58.625132 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:58.625469 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.626016 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.626762 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.626963 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.627049 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.629731 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:54507: BlockingRecv error: recv got EOF from 127.11.11.66:54507 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:58 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:58.774106 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:58.769228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:41855)
0928 12:58:58.769388 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:58.769401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:58.769506 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:58.769660 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:58.769662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:58.769824 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:58.769876 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.769877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.770777 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.770778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.771335 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:58.771337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.771669 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.771669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:58.771711 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.772230 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:58.772234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:58.774045 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
W20260928 12:58:58.774266 20481 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:58.769358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:41855 (local address 127.11.11.126:43429)
0928 12:58:58.769425 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:58.769427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:58.769457 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:58:58.769570 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:58.769570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:58.769583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:58.769633 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:58.769940 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.770691 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.771411 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:58.771593 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:58.771645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:58.774186 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:41855: BlockingRecv error: recv got EOF from 127.11.11.67:41855 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:59.520429 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:59.515699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:58051)
0928 12:58:59.515824 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:59.515835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:59.515894 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:59.516074 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:59.516075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:59.516263 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:59.516313 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.516315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.516979 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.516980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.517593 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.517595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.517948 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.517949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.517989 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.518411 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:59.518415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:59.520308 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
W20260928 12:58:59.520455 20485 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:59.515747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:58051 (local address 127.11.11.126:43429)
0928 12:58:59.515876 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:59.515878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:59.515889 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:58:59.515940 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:59.515941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:59.515951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:59.516021 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:59.516381 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.516874 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.517685 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.517882 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.517937 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.520390 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:58051: BlockingRecv error: recv got EOF from 127.11.11.65:58051 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:59.635351 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:59.630170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:48041)
0928 12:58:59.630368 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:59.630381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:59.630459 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:59.630646 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:59.630647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:59.630831 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:59.630883 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.630885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.631573 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.631574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.632155 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.632157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.632544 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.632546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.632600 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.633110 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:59.633115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:59.635246 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
W20260928 12:58:59.635414 20485 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:59.630285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:48041 (local address 127.11.11.126:43429)
0928 12:58:59.630382 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:59.630384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:59.630457 (+ 73us) server_negotiation.cc:407] Connection header received
0928 12:58:59.630536 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:59.630537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:59.630557 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:59.630621 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:59.630974 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.631483 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.632268 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.632467 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.632546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.635348 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:48041: BlockingRecv error: recv got EOF from 127.11.11.66:48041 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:58:59 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:58:59.779551 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:58:59.774723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:44889)
0928 12:58:59.774909 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:58:59.774923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:58:59.774994 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:58:59.775173 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:58:59.775174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:58:59.775330 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:58:59.775379 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.775380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.776306 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.776308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.776928 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:58:59.776930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.777267 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.777268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:58:59.777305 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.777781 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:58:59.777785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:58:59.779490 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
W20260928 12:58:59.779677 20485 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:58:59.774932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:44889 (local address 127.11.11.126:43429)
0928 12:58:59.775018 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:58:59.775020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:58:59.775030 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:58:59.775066 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:58:59.775067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:58:59.775080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:58:59.775134 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:58:59.775469 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.776182 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.777004 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:58:59.777196 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:58:59.777256 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:58:59.779586 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:44889: BlockingRecv error: recv got EOF from 127.11.11.67:44889 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:00.525827 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:00.520915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:47493)
0928 12:59:00.521046 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:00.521057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:00.521135 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:00.521413 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:00.521414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:00.521575 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:00.521624 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.521625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.522349 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.522350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.522879 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.522881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.523260 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.523261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.523301 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.523806 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:00.523811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:00.525753 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260928 12:59:00.526023 20489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:00.521133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47493 (local address 127.11.11.126:43429)
0928 12:59:00.521285 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:00.521287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:00.521295 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:59:00.521318 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:00.521318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:00.521329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:00.521400 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:00.521710 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.522275 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.522972 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.523189 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.523251 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.525911 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:47493: BlockingRecv error: recv got EOF from 127.11.11.65:47493 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:00.641136 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:00.636028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:49965)
0928 12:59:00.636214 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:00.636225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:00.636277 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:00.636441 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:00.636442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:00.636605 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:00.636656 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.636658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.637484 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.637484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.638013 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.638014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.638387 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.638387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.638427 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.638956 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:00.638961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:00.641074 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
W20260928 12:59:00.641233 20489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:00.636204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49965 (local address 127.11.11.126:43429)
0928 12:59:00.636268 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:00.636270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:00.636286 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:59:00.636337 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:00.636337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:00.636348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:00.636411 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:00.636737 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.637407 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.638111 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.638313 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.638379 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.641184 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:49965: BlockingRecv error: recv got EOF from 127.11.11.66:49965 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:00 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:00.785447 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:00.780354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:53185)
0928 12:59:00.780533 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:00.780592 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:00.780677 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:00.780836 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:00.780837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:00.781020 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:00.781071 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.781073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.781757 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.781758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.782299 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:00.782301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.782722 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.782723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:00.782796 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.783341 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:00.783349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:00.785375 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
W20260928 12:59:00.785586 20489 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:00.780413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:53185 (local address 127.11.11.126:43429)
0928 12:59:00.780489 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:00.780491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:00.780665 (+ 174us) server_negotiation.cc:407] Connection header received
0928 12:59:00.780740 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:00.780741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:00.780761 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:00.780829 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:00.781166 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.781665 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.782389 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:00.782630 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:00.782719 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:00.785501 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:53185: BlockingRecv error: recv got EOF from 127.11.11.67:53185 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:01.531185 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:01.526353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:51673)
0928 12:59:01.526502 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:01.526522 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:01.526576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:01.526768 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:01.526768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:01.526934 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:01.526984 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.526985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.527628 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.527628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.528192 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.528194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.528550 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.528551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.528596 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.529074 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:01.529078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:01.531120 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
W20260928 12:59:01.531299 20493 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:01.526478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:51673 (local address 127.11.11.126:43429)
0928 12:59:01.526635 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:01.526637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:01.526646 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:59:01.526671 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:01.526672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:01.526683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:01.526752 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:01.527053 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.527558 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.528265 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.528464 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.528522 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.531237 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:51673: BlockingRecv error: recv got EOF from 127.11.11.65:51673 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:01.646745 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:01.641682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:35139)
0928 12:59:01.641845 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:01.641863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:01.641933 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:01.642159 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:01.642159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:01.642338 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:01.642388 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.642391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.643102 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.643104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.643901 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.643903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.644279 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.644280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.644322 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.644834 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:01.644838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:01.646677 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260928 12:59:01.646876 20493 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:01.641764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:35139 (local address 127.11.11.126:43429)
0928 12:59:01.641852 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:01.641854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:01.641917 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:59:01.642026 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:01.642031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:01.642043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:01.642135 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:01.642457 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.642992 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.643979 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.644209 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.644265 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.646776 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:35139: BlockingRecv error: recv got EOF from 127.11.11.66:35139 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:01 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:01.791208 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:01.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:51479)
0928 12:59:01.786278 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:01.786291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:01.786392 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:01.786617 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:01.786619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:01.786800 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:01.786847 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.786849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.787514 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.787516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.788159 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:01.788161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.788552 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.788554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:01.788609 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.789116 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:01.789121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:01.791125 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260928 12:59:01.791332 20493 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:01.786159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51479 (local address 127.11.11.126:43429)
0928 12:59:01.786264 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:01.786266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:01.786372 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:59:01.786449 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:01.786450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:01.786466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:01.786601 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:01.786932 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.787428 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.788255 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:01.788446 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:01.788554 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:01.791245 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:51479: BlockingRecv error: recv got EOF from 127.11.11.67:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:02.543613 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:02.532004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:53283)
0928 12:59:02.532175 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:02.532188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:02.532257 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:02.532464 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:02.532464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:02.532659 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:02.532709 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.532711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.533737 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.533738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.534439 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.534442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.534820 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.534820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.534863 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.541634 (+ 6771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:02.541641 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:02.543549 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260928 12:59:02.543818 20497 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:02.532190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:53283 (local address 127.11.11.126:43429)
0928 12:59:02.532324 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:02.532326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:02.532336 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:59:02.532364 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:02.532365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:02.532376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:02.532447 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:02.532804 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.533630 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.534526 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.534759 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.534821 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.543733 (+ 8912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:53283: BlockingRecv error: recv got EOF from 127.11.11.65:53283 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:02.651770 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:02.647259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:50235)
0928 12:59:02.647392 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:02.647408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:02.647478 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:02.647600 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:02.647600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:02.647770 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:02.647824 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.647826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.648493 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.648494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.649042 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.649044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.649384 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.649385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.649425 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.649863 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:02.649868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:02.651709 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
W20260928 12:59:02.651939 20497 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:02.647358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:50235 (local address 127.11.11.126:43429)
0928 12:59:02.647405 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:02.647407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:02.647481 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:59:02.647538 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:02.647539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:02.647550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:02.647588 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:02.647886 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.648419 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.649127 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.649317 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.649370 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.651846 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:50235: BlockingRecv error: recv got EOF from 127.11.11.66:50235 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:02 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:02.796815 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:02.791813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:48893)
0928 12:59:02.791984 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:02.791996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:02.792135 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:02.792303 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:02.792304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:02.792497 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:02.792544 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.792546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.793249 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.793250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.793792 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:02.793794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.794160 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.794161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:02.794221 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.794753 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:02.794759 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:02.796696 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
W20260928 12:59:02.796852 20497 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:02.792003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:48893 (local address 127.11.11.126:43429)
0928 12:59:02.792072 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:02.792075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:02.792112 (+ 37us) server_negotiation.cc:407] Connection header received
0928 12:59:02.792196 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:02.792197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:02.792211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:02.792294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:02.792658 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.793153 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.793882 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:02.794081 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:02.794162 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:02.796785 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:48893: BlockingRecv error: recv got EOF from 127.11.11.67:48893 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:03.549458 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:03.544165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:35711)
0928 12:59:03.544316 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:03.544331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:03.544420 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:03.544680 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:03.544681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:03.544898 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:03.544950 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.544952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.545713 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.545715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.546332 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.546334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.546766 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.546767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.546833 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.547423 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:03.547429 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:03.549388 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260928 12:59:03.549616 20501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:03.544270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:35711 (local address 127.11.11.126:43429)
0928 12:59:03.544384 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:03.544386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:03.544394 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:59:03.544564 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:03.544566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:03.544579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:03.544671 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:03.545054 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.545615 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.546433 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.546677 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.546766 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.549508 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:35711: BlockingRecv error: recv got EOF from 127.11.11.65:35711 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:59:03.657554 20501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:03.652554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:34193 (local address 127.11.11.126:43429)
0928 12:59:03.652638 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:03.652640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:03.652651 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:59:03.652724 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:03.652725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:03.652744 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:03.652812 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:03.653160 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.653673 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.654430 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.654698 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.654795 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.657495 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:34193: BlockingRecv error: recv got EOF from 127.11.11.66:34193 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 12:59:03.657570 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:03.652407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:34193)
0928 12:59:03.652547 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:03.652560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:03.652648 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:03.652820 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:03.652821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:03.653020 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:03.653068 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.653070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.653777 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.653778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.654338 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.654340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.654793 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.654794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.654862 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.655422 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:03.655426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:03.657400 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:03 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:03.809062 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:03.797332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:44597)
0928 12:59:03.797500 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:03.797511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:03.797601 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:03.797764 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:03.797765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:03.797956 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:03.798000 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.798002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.798681 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.798682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.799246 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:03.799248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.799619 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.799620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:03.799678 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.806788 (+ 7110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:03.806793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:03.808957 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
W20260928 12:59:03.809131 20501 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:03.797404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:44597 (local address 127.11.11.126:43429)
0928 12:59:03.797471 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:03.797474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:03.797583 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:59:03.797655 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:03.797656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:03.797691 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:03.797756 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:03.798081 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.798579 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.799333 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:03.799532 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:03.799617 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:03.809068 (+ 9451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:44597: BlockingRecv error: recv got EOF from 127.11.11.67:44597 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:04.555272 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:04.550069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:34475)
0928 12:59:04.550249 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:04.550260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:04.550320 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:04.550479 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:04.550480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:04.550656 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:04.550719 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.550721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.551379 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.551380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.551948 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.551950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.552397 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.552398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.552456 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.553058 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:04.553063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:04.555210 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
W20260928 12:59:04.555397 20505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:04.550208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:34475 (local address 127.11.11.126:43429)
0928 12:59:04.550343 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:04.550345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:04.550356 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:59:04.550390 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:04.550390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:04.550401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:04.550464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:04.550789 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.551293 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.552094 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.552302 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.552362 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.555311 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:34475: BlockingRecv error: recv got EOF from 127.11.11.65:34475 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:04.663033 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:04.658154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:58209)
0928 12:59:04.658333 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:04.658345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:04.658414 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:04.658559 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:04.658559 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:04.658747 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:04.658795 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.658797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.659460 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.659461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.660016 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.660018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.660431 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.660432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.660480 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.660961 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:04.660966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:04.662920 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
W20260928 12:59:04.663080 20505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:04.658241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:58209 (local address 127.11.11.126:43429)
0928 12:59:04.658315 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:04.658317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:04.658418 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:59:04.658471 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:04.658472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:04.658484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:04.658527 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:04.658863 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.659361 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.660179 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.660366 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.660419 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.663014 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:58209: BlockingRecv error: recv got EOF from 127.11.11.66:58209 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:04 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:04.814050 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:04.809641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:55863)
0928 12:59:04.809811 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:04.809822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:04.809885 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:04.810014 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:04.810015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:04.810184 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:04.810228 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.810229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.810890 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.810892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.811414 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:04.811416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.811802 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.811803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:04.811844 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.812306 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:04.812310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:04.813984 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
W20260928 12:59:04.814208 20505 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:04.809748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:55863 (local address 127.11.11.126:43429)
0928 12:59:04.809817 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:04.809819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:04.809869 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:59:04.809940 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:04.809941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:04.809954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:04.810007 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:04.810290 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.810802 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.811520 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:04.811725 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:04.811791 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:04.814082 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:55863: BlockingRecv error: recv got EOF from 127.11.11.67:55863 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:05.561328 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:05.555941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:51573)
0928 12:59:05.556123 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:05.556172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:05.556272 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:05.556438 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:05.556439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:05.556619 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:05.556670 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.556671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.557742 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.557743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.558322 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.558325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.558753 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.558754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.558804 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.559299 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:05.559304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:05.561258 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":246,"threads_started":1}
W20260928 12:59:05.561452 20509 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:05.556076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:51573 (local address 127.11.11.126:43429)
0928 12:59:05.556205 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:05.556207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:05.556243 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:59:05.556334 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:05.556335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:05.556346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:05.556413 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:05.556785 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.557620 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.558414 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.558655 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.558721 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.561361 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:51573: BlockingRecv error: recv got EOF from 127.11.11.65:51573 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:05.669266 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:05.663743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:41757)
0928 12:59:05.663912 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:05.663927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:05.664002 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:05.664209 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:05.664210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:05.664374 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:05.664422 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.664424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.665208 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.665210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.665777 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.665779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.666179 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.666180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.666228 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.666884 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:05.666889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:05.669157 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
W20260928 12:59:05.669327 20509 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:05.663889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:41757 (local address 127.11.11.126:43429)
0928 12:59:05.663981 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:05.663983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:05.663992 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:59:05.664121 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:05.664122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:05.664132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:05.664186 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:05.664543 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.665103 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.665884 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.666096 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.666160 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.669253 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:41757: BlockingRecv error: recv got EOF from 127.11.11.66:41757 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:05 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:05.819239 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:05.814570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:43049)
0928 12:59:05.814713 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:05.814726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:05.814799 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:05.814947 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:05.814948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:05.815147 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:05.815192 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.815194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.815910 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.815911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.816482 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:05.816485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.816845 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.816846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:05.816903 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.817405 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:05.817410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:05.819167 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260928 12:59:05.819311 20509 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:05.814676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:43049 (local address 127.11.11.126:43429)
0928 12:59:05.814726 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:05.814728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:05.814780 (+ 52us) server_negotiation.cc:407] Connection header received
0928 12:59:05.814855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:05.814856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:05.814871 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:05.814940 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:05.815281 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.815811 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.816577 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:05.816769 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:05.816846 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:05.819262 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:43049: BlockingRecv error: recv got EOF from 127.11.11.67:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:06.567211 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:06.561969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:60829)
0928 12:59:06.562154 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:06.562169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:06.562248 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:06.562485 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:06.562486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:06.562700 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:06.562747 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.562749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.563476 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.563478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.564335 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.564338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.564710 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.564711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.564751 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.565285 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:06.565290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:06.567106 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":120,"threads_started":1}
W20260928 12:59:06.567266 20513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:06.562087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:60829 (local address 127.11.11.126:43429)
0928 12:59:06.562219 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:06.562221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:06.562232 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:59:06.562349 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:06.562351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:06.562372 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:06.562438 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:06.562824 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.563327 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.564407 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.564640 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.564698 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.567193 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:60829: BlockingRecv error: recv got EOF from 127.11.11.65:60829 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:06.675016 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:06.669854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:42245)
0928 12:59:06.670040 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:06.670071 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:06.670131 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:06.670305 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:06.670306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:06.670489 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:06.670547 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.670549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.671326 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.671327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.671871 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.671873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.672333 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.672334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.672400 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.672886 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:06.672891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:06.674916 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
W20260928 12:59:06.675071 20513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:06.670033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:42245 (local address 127.11.11.126:43429)
0928 12:59:06.670110 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:06.670113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:06.670144 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:59:06.670217 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:06.670218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:06.670232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:06.670290 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:06.670640 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.671235 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.671959 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.672254 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.672335 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.675008 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:42245: BlockingRecv error: recv got EOF from 127.11.11.66:42245 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:06 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.67@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:06.824580 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:06.819839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.67:58869)
0928 12:59:06.820010 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:06.820021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:06.820130 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:06.820327 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:06.820328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:06.820484 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:06.820530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.820532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.821192 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.821193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.821826 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:06.821828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.822162 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.822163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:06.822203 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.822700 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:06.822704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:06.824496 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
W20260928 12:59:06.824733 20513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:06.819912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:58869 (local address 127.11.11.126:43429)
0928 12:59:06.819994 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:06.819996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:06.820126 (+ 130us) server_negotiation.cc:407] Connection header received
0928 12:59:06.820231 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:06.820232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:06.820253 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:06.820306 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:06.820599 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.821114 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.821901 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:06.822093 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:06.822152 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:06.824640 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.67:58869: BlockingRecv error: recv got EOF from 127.11.11.67:58869 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 12:59:07 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:07 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.65@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:07.579438 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:07.567835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.65:44065)
0928 12:59:07.567977 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:07.567993 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:07.568083 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:07.568386 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:07.568387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:07.568629 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:07.568711 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:07.568713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.569520 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.569521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:07.570130 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:07.570132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.570536 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.570537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:07.570578 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:07.577283 (+ 6705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:07.577289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:07.579333 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260928 12:59:07.579581 20517 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:07.567968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:44065 (local address 127.11.11.126:43429)
0928 12:59:07.568103 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:07.568106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:07.568116 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:59:07.568200 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:07.568201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:07.568216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:07.568312 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:07.568799 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.569438 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.570228 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.570467 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.570527 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:07.579488 (+ 8961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.65:44065: BlockingRecv error: recv got EOF from 127.11.11.65:44065 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"spinlock_wait_cycles":1792,"thread_start_us":107,"threads_started":1}
Sep 28 12:59:07 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:59:07 dist-test-slave-8sjz krb5kdc[19459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.11.66@KRBTEST.COM for oryx/127.11.11.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:59:07.680143 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:07.675635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43429 (local address 127.11.11.66:45137)
0928 12:59:07.675771 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:07.675789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:07.675850 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:07.676000 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:07.676000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:07.676186 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:07.676237 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:07.676238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.676855 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.676855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:07.677386 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:07.677389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.677706 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.677707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:07.677747 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:07.678272 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:07.678276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:07.680011 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.11.126:43429: Server oryx/127.11.11.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
W20260928 12:59:07.680269 20517 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:07.675726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:45137 (local address 127.11.11.126:43429)
0928 12:59:07.675822 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:07.675824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:07.675836 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:59:07.675914 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:07.675915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:07.675925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:07.675979 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:07.676297 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.676786 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.677458 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.677641 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.677714 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:07.680171 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.11.66:45137: BlockingRecv error: recv got EOF from 127.11.11.66:45137 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 12:59:07.732055 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20042
I20260928 12:59:07.737212 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20194
I20260928 12:59:07.742017 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20327
I20260928 12:59:07.746425 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 19972
2026-09-28T12:59:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17531 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T12:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:59:07Z Disabled control of system clock
I20260928 12:59:07.775230 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41541
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:59:07.857985 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:07.858227 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:07.858248 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:07.859690 20522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:59:07.859745 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:07.859757 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:07.859771 20522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:59:07.859784 20522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:59:07.861260 20522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:07.861554 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:07.861882 20522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:07.867307 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:07.867340 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:07.867301 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:07.867395 20522 server_base.cc:1034] running on GCE node
I20260928 12:59:07.867812 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:07.868191 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:07.869365 20522 hybrid_clock.cc:648] HybridClock initialized: now 1790600347869350 us; error 47 us; skew 500 ppm
I20260928 12:59:07.870651 20522 webserver.cc:460] Webserver started at http://127.11.11.126:34637/ using document root <none> and password file <none>
I20260928 12:59:07.870898 20522 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:07.870966 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:07.871130 20522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:07.872331 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "7800d429b31b4f22929738829e9d5426"
format_stamp: "Formatted at 2026-09-28 12:59:07 on dist-test-slave-8sjz"
server_key: "7000daa890b9b638e215835e23172bdc"
server_key_iv: "31760ae05f374b2d053efe113ac74059"
server_key_version: "encryptionkey@0"
I20260928 12:59:07.872705 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "7800d429b31b4f22929738829e9d5426"
format_stamp: "Formatted at 2026-09-28 12:59:07 on dist-test-slave-8sjz"
server_key: "7000daa890b9b638e215835e23172bdc"
server_key_iv: "31760ae05f374b2d053efe113ac74059"
server_key_version: "encryptionkey@0"
I20260928 12:59:07.874001 20522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:07.874748 20536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:07.874909 20522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:59:07.875010 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "7800d429b31b4f22929738829e9d5426"
format_stamp: "Formatted at 2026-09-28 12:59:07 on dist-test-slave-8sjz"
server_key: "7000daa890b9b638e215835e23172bdc"
server_key_iv: "31760ae05f374b2d053efe113ac74059"
server_key_version: "encryptionkey@0"
I20260928 12:59:07.875092 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:07.885740 20522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:07.886097 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:07.886258 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:07.890239 20522 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41541
I20260928 12:59:07.890275 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41541 every 8 connection(s)
I20260928 12:59:07.890650 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:59:07.891350 20589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:07.894023 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426: Bootstrap starting.
I20260928 12:59:07.894665 20589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:07.894974 20589 log.cc:824] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:07.895679 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426: No bootstrap required, opened a new log
I20260928 12:59:07.897301 20589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } }
I20260928 12:59:07.897444 20589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:07.897467 20589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7800d429b31b4f22929738829e9d5426, State: Initialized, Role: FOLLOWER
I20260928 12:59:07.897656 20589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } }
I20260928 12:59:07.897723 20589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:59:07.897745 20589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:59:07.897816 20589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:07.898357 20589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } }
I20260928 12:59:07.898471 20589 leader_election.cc:302] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7800d429b31b4f22929738829e9d5426; no voters:
I20260928 12:59:07.898674 20589 leader_election.cc:288] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:59:07.898725 20592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:07.898927 20592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [term 1 LEADER]: Becoming Leader. State: Replica: 7800d429b31b4f22929738829e9d5426, State: Running, Role: LEADER
I20260928 12:59:07.898950 20589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:59:07.899055 20592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } }
I20260928 12:59:07.899456 20594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7800d429b31b4f22929738829e9d5426. Latest consensus state: current_term: 1 leader_uuid: "7800d429b31b4f22929738829e9d5426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } } }
I20260928 12:59:07.899611 20594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:07.899590 20593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7800d429b31b4f22929738829e9d5426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7800d429b31b4f22929738829e9d5426" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41541 } } }
I20260928 12:59:07.899771 20593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426 [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:07.899848 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20522
I20260928 12:59:07.899938 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:59:07.900076 20601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:59:07.900202 11309 external_mini_cluster.cc:1428] Setting key 5a2af082ba939c12c83fa974093d01f6
I20260928 12:59:07.900599 20601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:59:07.902254 20601 catalog_manager.cc:1347] Generated new cluster ID: 0e1a8cdcde784694bccd8c6811e50b3e
I20260928 12:59:07.902309 20601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:59:07.906843 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:07.900815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.11.11.126:41541)
0928 12:59:07.901225 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:07.901235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:07.901267 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:59:07.901388 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:07.901395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:07.901464 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:07.901592 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:07.901947 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.903307 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.904446 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:07.905116 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:07.905247 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:07.905991 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:07.905995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:07.905996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:07.906220 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:07.906264 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:07.906280 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:07.906352 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:07.906425 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:59:07.906612 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":59,"threads_started":1}
I20260928 12:59:07.928514 20601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:59:07.929291 20601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:59:07.933012 20601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7800d429b31b4f22929738829e9d5426: Generated new TSK 0
I20260928 12:59:07.933230 20601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:59:07.937850 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--builtin_ntp_servers=127.11.11.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:59:08.022917 20613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.023171 20613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.023211 20613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.024560 20613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.024633 20613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.024678 20613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:59:08.026225 20613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.026531 20613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.026880 20613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.032150 20619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.032178 20613 server_base.cc:1034] running on GCE node
W20260928 12:59:08.032150 20621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.032142 20618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.032621 20613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.032904 20613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.034070 20613 hybrid_clock.cc:648] HybridClock initialized: now 1790600348034064 us; error 44 us; skew 500 ppm
I20260928 12:59:08.035487 20613 webserver.cc:460] Webserver started at http://127.11.11.65:42243/ using document root <none> and password file <none>
I20260928 12:59:08.035797 20613 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.035874 20613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.036073 20613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.037225 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "c730d29608cf4232af6dc45d933cd06e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "e291d540935515a4e1baa673e4f16b60"
server_key_iv: "97f6e21e184a98191fd2a29f1d7206e8"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.037601 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "c730d29608cf4232af6dc45d933cd06e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "e291d540935515a4e1baa673e4f16b60"
server_key_iv: "97f6e21e184a98191fd2a29f1d7206e8"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.039031 20613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:59:08.039891 20627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.040247 20613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:08.040354 20613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "c730d29608cf4232af6dc45d933cd06e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "e291d540935515a4e1baa673e4f16b60"
server_key_iv: "97f6e21e184a98191fd2a29f1d7206e8"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.040446 20613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.054287 20613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.054648 20613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.054809 20613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.055056 20613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.055413 20613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.055469 20613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.055512 20613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.055538 20613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.061342 20613 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:34857
I20260928 12:59:08.061422 20740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:34857 every 8 connection(s)
I20260928 12:59:08.061781 20613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:59:08.063022 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20613
I20260928 12:59:08.063130 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:59:08.063325 11309 external_mini_cluster.cc:1428] Setting key c8bbff6ab97f3f8ecb908c59cedb414a
I20260928 12:59:08.064455 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--builtin_ntp_servers=127.11.11.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 12:59:08.068991 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.062669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:53141 (local address 127.11.11.126:41541)
0928 12:59:08.062741 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.062743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.063054 (+ 311us) server_negotiation.cc:407] Connection header received
0928 12:59:08.063823 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.063824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.063839 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.063889 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.064856 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.065511 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.067105 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.067422 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.067502 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.068565 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.068567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.068567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.068574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.068593 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.068597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.068693 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.068814 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.068874 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:59:08.069190 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.062649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41541 (local address 127.11.11.65:53141)
0928 12:59:08.062891 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.062940 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.063719 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.063978 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.063982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.063999 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.064746 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.064753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.065649 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.065650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.067001 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.067004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.067516 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.067518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.067692 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.068383 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.068383 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.068425 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.068457 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.068694 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.068701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.068704 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.068814 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.068939 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260928 12:59:08.069902 20741 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41541
I20260928 12:59:08.069994 20741 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.070209 20741 heartbeater.cc:511] Master 127.11.11.126:41541 requested a full tablet report, sending...
I20260928 12:59:08.070770 20553 ts_manager.cc:194] Registered new tserver with Master: c730d29608cf4232af6dc45d933cd06e (127.11.11.65:34857)
I20260928 12:59:08.071419 20553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.65:53141
W20260928 12:59:08.149587 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.149847 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.149885 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.151115 20744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.151185 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.151230 20744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:59:08.152668 20744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.152937 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.153256 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:59:08.158349 20744 server_base.cc:1034] running on GCE node
W20260928 12:59:08.158257 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.158277 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.158257 20752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.158715 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.158928 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.160111 20744 hybrid_clock.cc:648] HybridClock initialized: now 1790600348160092 us; error 38 us; skew 500 ppm
I20260928 12:59:08.161346 20744 webserver.cc:460] Webserver started at http://127.11.11.66:43707/ using document root <none> and password file <none>
I20260928 12:59:08.161586 20744 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.161633 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.161795 20744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.162892 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "096a1e1e4407486ebbdbe78b88ba5927"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "d8c95b959cec82c31bf26d14d94596f5"
server_key_iv: "16859d7e843b7e053a2bb70cf0ae68ab"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.163264 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "096a1e1e4407486ebbdbe78b88ba5927"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "d8c95b959cec82c31bf26d14d94596f5"
server_key_iv: "16859d7e843b7e053a2bb70cf0ae68ab"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.164680 20744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:59:08.165374 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.165552 20744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:59:08.165678 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "096a1e1e4407486ebbdbe78b88ba5927"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "d8c95b959cec82c31bf26d14d94596f5"
server_key_iv: "16859d7e843b7e053a2bb70cf0ae68ab"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.165751 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.183272 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.183627 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.183789 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.184120 20744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.184479 20744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.184514 20744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.184536 20744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.184549 20744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.190378 20744 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:41259
I20260928 12:59:08.190429 20871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:41259 every 8 connection(s)
I20260928 12:59:08.190805 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:59:08.196516 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.191679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:56129 (local address 127.11.11.126:41541)
0928 12:59:08.191783 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.191785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.192015 (+ 230us) server_negotiation.cc:407] Connection header received
0928 12:59:08.192608 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.192609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.192622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.192672 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.193367 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.193908 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.194913 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.195122 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.195204 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.196189 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.196191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.196191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.196196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.196213 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.196223 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.196283 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.196389 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.196421 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 12:59:08.196760 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.191565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41541 (local address 127.11.11.66:56129)
0928 12:59:08.191858 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.191913 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.192488 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.192757 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.192764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.192777 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.193267 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.193271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.194018 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.194020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.194817 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.194819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.195213 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.195214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.195354 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.195944 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.195944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.195998 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.196023 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.196298 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.196304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.196307 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.196394 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.196510 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260928 12:59:08.197304 20872 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41541
I20260928 12:59:08.197388 20872 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.197643 20872 heartbeater.cc:511] Master 127.11.11.126:41541 requested a full tablet report, sending...
I20260928 12:59:08.198040 20553 ts_manager.cc:194] Registered new tserver with Master: 096a1e1e4407486ebbdbe78b88ba5927 (127.11.11.66:41259)
I20260928 12:59:08.198478 20553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.66:56129
I20260928 12:59:08.201049 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20744
I20260928 12:59:08.201150 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:59:08.201354 11309 external_mini_cluster.cc:1428] Setting key f2e371bfb6c6a8e931d8473ef36fbcdf
I20260928 12:59:08.202286 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--builtin_ntp_servers=127.11.11.84:38253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:59:08.285377 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.285645 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.285681 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.287002 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.287068 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.287110 20875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:59:08.288546 20875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.288825 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.289140 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.294147 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.294194 20875 server_base.cc:1034] running on GCE node
W20260928 12:59:08.294231 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.294181 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.294627 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.294836 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.295974 20875 hybrid_clock.cc:648] HybridClock initialized: now 1790600348295953 us; error 36 us; skew 500 ppm
I20260928 12:59:08.297250 20875 webserver.cc:460] Webserver started at http://127.11.11.67:33751/ using document root <none> and password file <none>
I20260928 12:59:08.297503 20875 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.297549 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.297644 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.298761 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "a7f6d0adaf404650a23bfcee08a7a717"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "93948a1326862da24c3ec2897699b803"
server_key_iv: "81bc72b93c79d7133f76e224e86d7bca"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.299202 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "a7f6d0adaf404650a23bfcee08a7a717"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "93948a1326862da24c3ec2897699b803"
server_key_iv: "81bc72b93c79d7133f76e224e86d7bca"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.300683 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:08.301435 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.301671 20875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:08.301769 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "a7f6d0adaf404650a23bfcee08a7a717"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "93948a1326862da24c3ec2897699b803"
server_key_iv: "81bc72b93c79d7133f76e224e86d7bca"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.301842 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.312760 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.313117 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.313272 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.313617 20875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.314037 20875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.314077 20875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.314102 20875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.314116 20875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.320200 20875 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40057
I20260928 12:59:08.320297 21002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40057 every 8 connection(s)
I20260928 12:59:08.320647 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:59:08.326139 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.321454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:59919 (local address 127.11.11.126:41541)
0928 12:59:08.321506 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.321509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.321797 (+ 288us) server_negotiation.cc:407] Connection header received
0928 12:59:08.322331 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.322332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.322345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.322417 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.323085 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.323639 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.324789 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.324985 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.325050 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.325930 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.325931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.325931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.325935 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.325946 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.325949 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.325986 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.326051 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.326075 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:59:08.326355 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.321465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41541 (local address 127.11.11.67:59919)
0928 12:59:08.321669 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.321706 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.322211 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.322473 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.322477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.322485 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.323007 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.323011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.323755 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.323756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.324703 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:08.324706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:08.325065 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:08.325066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:08.325177 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:08.325803 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.325804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.325833 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.325858 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.325988 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.325991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.325993 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.326043 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.326123 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260928 12:59:08.326902 21003 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41541
I20260928 12:59:08.327013 21003 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.327201 21003 heartbeater.cc:511] Master 127.11.11.126:41541 requested a full tablet report, sending...
I20260928 12:59:08.327418 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 20875
I20260928 12:59:08.327523 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:59:08.327652 20553 ts_manager.cc:194] Registered new tserver with Master: a7f6d0adaf404650a23bfcee08a7a717 (127.11.11.67:40057)
I20260928 12:59:08.327737 11309 external_mini_cluster.cc:1428] Setting key b9bea0390cac07886614e8a35cb39229
I20260928 12:59:08.328109 20553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.67:59919
I20260928 12:59:08.328739 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:59:08.330590 20610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:08.330137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.11.11.126:41541)
0928 12:59:08.330205 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.330207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.330310 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:59:08.330358 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.330359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.330369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.330424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.330536 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58648: BlockingRecv error: recv got EOF from 127.0.0.1:58648 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:59:08.330639 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20613
I20260928 12:59:08.335908 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20744
I20260928 12:59:08.340633 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20875
I20260928 12:59:08.345047 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 20522
2026-09-28T12:59:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (596 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T12:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:59:08Z Disabled control of system clock
I20260928 12:59:08.372282 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37533
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:37533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:59:08.453538 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.453756 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.453778 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.454871 21014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:59:08.454903 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.454917 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.454929 21014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:59:08.454942 21014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:59:08.456339 21014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:37533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.456570 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.456816 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.461712 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.461750 21014 server_base.cc:1034] running on GCE node
W20260928 12:59:08.461730 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.461719 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.462183 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.462504 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.463685 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790600348463665 us; error 36 us; skew 500 ppm
I20260928 12:59:08.465070 21014 webserver.cc:460] Webserver started at http://127.11.11.126:33795/ using document root <none> and password file <none>
I20260928 12:59:08.465332 21014 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.465402 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.465557 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.466724 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "70f41ab2b238460daff45a40f3f01d3e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "ebe8031cf00bc6d846adfe49db06de43"
server_key_iv: "c9a645918eab95a76973172df67938ba"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.467082 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "70f41ab2b238460daff45a40f3f01d3e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "ebe8031cf00bc6d846adfe49db06de43"
server_key_iv: "c9a645918eab95a76973172df67938ba"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.468427 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:59:08.469163 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.469337 21014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:59:08.469440 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "70f41ab2b238460daff45a40f3f01d3e"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "ebe8031cf00bc6d846adfe49db06de43"
server_key_iv: "c9a645918eab95a76973172df67938ba"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.469518 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.472949 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.473219 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.473361 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.477941 21014 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:37533
I20260928 12:59:08.477972 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:37533 every 8 connection(s)
I20260928 12:59:08.478379 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:59:08.479075 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:08.481966 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e: Bootstrap starting.
I20260928 12:59:08.482556 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:08.482864 21081 log.cc:824] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:08.483551 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e: No bootstrap required, opened a new log
I20260928 12:59:08.485257 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } }
I20260928 12:59:08.485402 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:08.485425 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f41ab2b238460daff45a40f3f01d3e, State: Initialized, Role: FOLLOWER
I20260928 12:59:08.485584 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } }
I20260928 12:59:08.485666 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:59:08.485688 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:59:08.485737 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:08.486305 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } }
I20260928 12:59:08.486418 21081 leader_election.cc:302] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f41ab2b238460daff45a40f3f01d3e; no voters:
I20260928 12:59:08.486619 21081 leader_election.cc:288] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:59:08.486644 21084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:08.486645 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21014
I20260928 12:59:08.486755 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:59:08.486757 21084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [term 1 LEADER]: Becoming Leader. State: Replica: 70f41ab2b238460daff45a40f3f01d3e, State: Running, Role: LEADER
I20260928 12:59:08.486831 21084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } }
I20260928 12:59:08.486894 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:59:08.486953 11309 external_mini_cluster.cc:1428] Setting key c1c22936da21ecf26c87d463f12cf469
I20260928 12:59:08.487170 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f41ab2b238460daff45a40f3f01d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } } }
I20260928 12:59:08.487684 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:08.487605 21086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f41ab2b238460daff45a40f3f01d3e. Latest consensus state: current_term: 1 leader_uuid: "70f41ab2b238460daff45a40f3f01d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f41ab2b238460daff45a40f3f01d3e" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37533 } } }
I20260928 12:59:08.487942 21096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:59:08.487944 21086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:08.488562 21096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:59:08.490284 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.487648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.11.11.126:37533)
0928 12:59:08.487885 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.487895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.488215 (+ 320us) server_negotiation.cc:407] Connection header received
0928 12:59:08.488349 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.488357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.488409 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.488557 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.489759 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.489762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.489763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.489767 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.489815 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.489833 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.489901 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.490026 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.490099 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260928 12:59:08.490432 21096 catalog_manager.cc:1347] Generated new cluster ID: f09386a4b3de4979816c854221879891
I20260928 12:59:08.490476 21096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:59:08.500821 21096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:59:08.501031 21096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:59:08.503620 21096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f41ab2b238460daff45a40f3f01d3e: Generated new TSK 0
I20260928 12:59:08.503862 21096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:59:08.507488 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--builtin_ntp_servers=127.11.11.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:59:08.602432 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.602754 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.602797 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.604135 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.604199 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.604243 21105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:59:08.605644 21105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.605923 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.606232 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.612265 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.612279 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.612627 21105 server_base.cc:1034] running on GCE node
W20260928 12:59:08.612377 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.612967 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.613240 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.614403 21105 hybrid_clock.cc:648] HybridClock initialized: now 1790600348614389 us; error 32 us; skew 500 ppm
I20260928 12:59:08.615765 21105 webserver.cc:460] Webserver started at http://127.11.11.65:39097/ using document root <none> and password file <none>
I20260928 12:59:08.616086 21105 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.616168 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.616318 21105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.617506 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "e57ca11fd69e485395f9baeab45523fa"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "9fc5487282f70fb07004120dd2e16b35"
server_key_iv: "a3f95d82c45e1254c1093c6ead8e537c"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.617887 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "e57ca11fd69e485395f9baeab45523fa"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "9fc5487282f70fb07004120dd2e16b35"
server_key_iv: "a3f95d82c45e1254c1093c6ead8e537c"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.619272 21105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:59:08.620088 21119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.620266 21105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:08.620348 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "e57ca11fd69e485395f9baeab45523fa"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "9fc5487282f70fb07004120dd2e16b35"
server_key_iv: "a3f95d82c45e1254c1093c6ead8e537c"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.620422 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.623878 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.624205 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.624346 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.624608 21105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.624949 21105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.624981 21105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.625031 21105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.625056 21105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.632128 21105 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:39729
I20260928 12:59:08.632164 21232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:39729 every 8 connection(s)
I20260928 12:59:08.632513 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:59:08.633138 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21105
I20260928 12:59:08.633219 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:59:08.633409 11309 external_mini_cluster.cc:1428] Setting key b5ef6258a8dd259a5a2e3827f8cb411f
I20260928 12:59:08.634481 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--builtin_ntp_servers=127.11.11.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 12:59:08.635912 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.633255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:51099 (local address 127.11.11.126:37533)
0928 12:59:08.633305 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.633308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.633623 (+ 315us) server_negotiation.cc:407] Connection header received
0928 12:59:08.634267 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.634268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.634276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.634313 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.635670 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.635672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.635672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.635677 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.635693 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.635698 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.635750 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.635825 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.635843 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:59:08.636181 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.633276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37533 (local address 127.11.11.65:51099)
0928 12:59:08.633487 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.633530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.634178 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.634400 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.634404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.634413 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.635506 (+ 1093us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.635506 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.635537 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.635568 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.635748 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.635753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.635758 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.635829 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.635895 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260928 12:59:08.636732 21233 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37533
I20260928 12:59:08.636861 21233 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.637023 21233 heartbeater.cc:511] Master 127.11.11.126:37533 requested a full tablet report, sending...
I20260928 12:59:08.637626 21045 ts_manager.cc:194] Registered new tserver with Master: e57ca11fd69e485395f9baeab45523fa (127.11.11.65:39729)
W20260928 12:59:08.717582 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.717810 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.717833 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.718973 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.719009 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.719034 21236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:59:08.720357 21236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.720564 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.720815 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.725904 21244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.725929 21242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.725983 21241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.726231 21236 server_base.cc:1034] running on GCE node
I20260928 12:59:08.726441 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.726646 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.727792 21236 hybrid_clock.cc:648] HybridClock initialized: now 1790600348727779 us; error 28 us; skew 500 ppm
I20260928 12:59:08.729259 21236 webserver.cc:460] Webserver started at http://127.11.11.66:46337/ using document root <none> and password file <none>
I20260928 12:59:08.729508 21236 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.729558 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.729729 21236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.730844 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "672cfc755e2448b3bae39fbb2a3c2278"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "0539ed5c0e2f0757cd99615b947abb01"
server_key_iv: "eae8fd2a5c903b14f62f12771ea02c6e"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.731226 21236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "672cfc755e2448b3bae39fbb2a3c2278"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "0539ed5c0e2f0757cd99615b947abb01"
server_key_iv: "eae8fd2a5c903b14f62f12771ea02c6e"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.732693 21236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:59:08.733439 21250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.733621 21236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.733711 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "672cfc755e2448b3bae39fbb2a3c2278"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "0539ed5c0e2f0757cd99615b947abb01"
server_key_iv: "eae8fd2a5c903b14f62f12771ea02c6e"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.733801 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.737356 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.737629 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.737766 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.737970 21236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.738278 21236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.738312 21236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.738332 21236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.738376 21236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.744251 21236 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:35795
I20260928 12:59:08.744326 21363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:35795 every 8 connection(s)
I20260928 12:59:08.744663 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:59:08.747574 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.745416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:41971 (local address 127.11.11.126:37533)
0928 12:59:08.745482 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.745484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.745714 (+ 230us) server_negotiation.cc:407] Connection header received
0928 12:59:08.746164 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.746165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.746170 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.746223 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.747356 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.747357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.747358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.747362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.747376 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.747380 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.747428 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.747506 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.747519 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:59:08.747787 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.745409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37533 (local address 127.11.11.66:41971)
0928 12:59:08.745584 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.745624 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.746078 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.746279 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.746282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.746290 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.747211 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.747211 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.747241 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.747270 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.747437 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.747441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.747444 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.747503 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.747552 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260928 12:59:08.748371 21364 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37533
I20260928 12:59:08.748473 21364 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.748643 21364 heartbeater.cc:511] Master 127.11.11.126:37533 requested a full tablet report, sending...
I20260928 12:59:08.749009 21045 ts_manager.cc:194] Registered new tserver with Master: 672cfc755e2448b3bae39fbb2a3c2278 (127.11.11.66:35795)
I20260928 12:59:08.750130 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21236
I20260928 12:59:08.750226 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:59:08.750411 11309 external_mini_cluster.cc:1428] Setting key 2f13c77624052d7de7b34b71be50912b
I20260928 12:59:08.751298 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--builtin_ntp_servers=127.11.11.84:42625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:59:08.833491 21367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:08.833777 21367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:08.833820 21367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:08.835032 21367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:08.835108 21367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:08.835148 21367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:59:08.836706 21367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:42625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:08.837010 21367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:08.837334 21367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:08.842635 21372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.842630 21373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:08.842656 21375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:08.842911 21367 server_base.cc:1034] running on GCE node
I20260928 12:59:08.843219 21367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:08.843509 21367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:08.844688 21367 hybrid_clock.cc:648] HybridClock initialized: now 1790600348844691 us; error 49 us; skew 500 ppm
I20260928 12:59:08.846159 21367 webserver.cc:460] Webserver started at http://127.11.11.67:41687/ using document root <none> and password file <none>
I20260928 12:59:08.846431 21367 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:08.846508 21367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:08.846680 21367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:08.847800 21367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "85e1de9c083b4d8caaf4fab589059ede"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "fbeeb6c7a26df20ab89c0c4f1c1839a3"
server_key_iv: "3831a4907c4adb3da078b2d2c9a67b02"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.848276 21367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "85e1de9c083b4d8caaf4fab589059ede"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "fbeeb6c7a26df20ab89c0c4f1c1839a3"
server_key_iv: "3831a4907c4adb3da078b2d2c9a67b02"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.849599 21367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:59:08.850301 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.850467 21367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:59:08.850575 21367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "85e1de9c083b4d8caaf4fab589059ede"
format_stamp: "Formatted at 2026-09-28 12:59:08 on dist-test-slave-8sjz"
server_key: "fbeeb6c7a26df20ab89c0c4f1c1839a3"
server_key_iv: "3831a4907c4adb3da078b2d2c9a67b02"
server_key_version: "encryptionkey@0"
I20260928 12:59:08.850657 21367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:08.854306 21367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:08.854597 21367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:08.854748 21367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:08.854952 21367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:08.855279 21367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:08.855334 21367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.855376 21367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:08.855403 21367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:08.861375 21367 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:44259
I20260928 12:59:08.861418 21494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:44259 every 8 connection(s)
I20260928 12:59:08.861793 21367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:59:08.864982 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.862583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51959 (local address 127.11.11.126:37533)
0928 12:59:08.862633 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.862635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.862879 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:59:08.863389 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.863391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.863398 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.863472 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.864760 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:08.864762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:08.864762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:59:08.864766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:08.864781 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:59:08.864785 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:08.864844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:08.864909 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:59:08.864922 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:59:08.865192 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:08.862567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37533 (local address 127.11.11.67:51959)
0928 12:59:08.862751 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:08.862786 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:08.863257 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:08.863545 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:08.863549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:08.863558 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:08.864598 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:59:08.864598 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:08.864636 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:59:08.864666 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:08.864843 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:08.864846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:08.864848 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:59:08.864902 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:59:08.864951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 12:59:08.865751 21495 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37533
I20260928 12:59:08.865834 21495 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:08.865998 21495 heartbeater.cc:511] Master 127.11.11.126:37533 requested a full tablet report, sending...
I20260928 12:59:08.866001 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21367
I20260928 12:59:08.866093 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:59:08.866318 11309 external_mini_cluster.cc:1428] Setting key d1c49ced8847d82092b6266536321389
I20260928 12:59:08.866389 21045 ts_manager.cc:194] Registered new tserver with Master: 85e1de9c083b4d8caaf4fab589059ede (127.11.11.67:44259)
I20260928 12:59:08.867172 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:59:08.868872 21095 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:59:08.868460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.11.11.126:37533)
0928 12:59:08.868513 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:08.868515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:08.868617 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:59:08.868672 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:08.868673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:08.868677 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:08.868724 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:08.868827 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43366: BlockingRecv error: recv got EOF from 127.0.0.1:43366 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:59:08.869011 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21105
I20260928 12:59:08.874900 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21236
I20260928 12:59:08.879698 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21367
I20260928 12:59:08.883955 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21014
2026-09-28T12:59:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (539 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:59:08 dist-test-slave-8sjz krb5kdc[21505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:59:08 dist-test-slave-8sjz krb5kdc[21505](info): set up 2 sockets
Sep 28 12:59:08 dist-test-slave-8sjz krb5kdc[21505](info): commencing operation
krb5kdc: starting...
W20260928 12:59:09.918143 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:59:10.047243 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37837
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:10.133132 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:10.133347 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:10.133369 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:10.134482 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:59:10.134519 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:10.134534 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:10.134546 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:59:10.134558 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:59:10.135962 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:10.136184 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:10.136368 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:10.141198 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.141201 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.141230 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.141767 21518 server_base.cc:1034] running on GCE node
I20260928 12:59:10.142040 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:10.142328 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:10.143500 21518 hybrid_clock.cc:648] HybridClock initialized: now 1790600350143528 us; error 107 us; skew 500 ppm
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:10.146101 21518 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:59:10.146597 21518 webserver.cc:460] Webserver started at http://127.11.11.126:34579/ using document root <none> and password file <none>
I20260928 12:59:10.146868 21518 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:10.146940 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:10.147094 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:10.148268 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "7735ff5b04e0757ace02aa9db98f20b5"
server_key_iv: "7035585c8dc4d6837e5d59541cee24ef"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.148638 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "7735ff5b04e0757ace02aa9db98f20b5"
server_key_iv: "7035585c8dc4d6837e5d59541cee24ef"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.149986 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:59:10.150720 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.150887 21518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:59:10.150996 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "7735ff5b04e0757ace02aa9db98f20b5"
server_key_iv: "7035585c8dc4d6837e5d59541cee24ef"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.151070 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:10.166515 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:10.168526 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:10.168705 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:10.172771 21518 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:37837
I20260928 12:59:10.172808 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:37837 every 8 connection(s)
I20260928 12:59:10.173171 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:59:10.174175 21586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:10.176625 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: Bootstrap starting.
I20260928 12:59:10.177224 21586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:10.177524 21586 log.cc:824] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:10.178289 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: No bootstrap required, opened a new log
I20260928 12:59:10.179947 21586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } }
I20260928 12:59:10.180172 21586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:10.180203 21586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b94eeab2eb44e6c89dd2e9e9f42bf8a, State: Initialized, Role: FOLLOWER
I20260928 12:59:10.180389 21586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } }
I20260928 12:59:10.180454 21586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:59:10.180509 21586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:59:10.180558 21586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:10.181444 21586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } }
I20260928 12:59:10.181543 21586 leader_election.cc:302] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b94eeab2eb44e6c89dd2e9e9f42bf8a; no voters:
I20260928 12:59:10.181743 21586 leader_election.cc:288] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:59:10.181820 21589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:10.182008 21589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [term 1 LEADER]: Becoming Leader. State: Replica: 9b94eeab2eb44e6c89dd2e9e9f42bf8a, State: Running, Role: LEADER
I20260928 12:59:10.182063 21586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:59:10.182188 21589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } }
I20260928 12:59:10.182585 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } } }
I20260928 12:59:10.182790 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:10.182595 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b94eeab2eb44e6c89dd2e9e9f42bf8a. Latest consensus state: current_term: 1 leader_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b94eeab2eb44e6c89dd2e9e9f42bf8a" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 37837 } } }
I20260928 12:59:10.183015 21598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:59:10.183063 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:10.183339 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21518
I20260928 12:59:10.183430 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:59:10.183653 11309 external_mini_cluster.cc:1428] Setting key 5d1fd5712eca5f50e42880b793a50a9f
I20260928 12:59:10.184163 21598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:59:10.185947 21598 catalog_manager.cc:1347] Generated new cluster ID: 3ff15269ea02455caf22c57861ac96e2
I20260928 12:59:10.186010 21598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:10.195757 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.184171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.11.11.126:37837)
0928 12:59:10.184499 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.184509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.184540 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:59:10.184820 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.184830 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.184901 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.185050 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:10.185726 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.187150 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.188213 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.188859 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.188986 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.191872 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:10.191877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:10.191877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:10.191890 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:10.193958 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.194405 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.194408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.194409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.194519 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.194732 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.194732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.194732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.195055 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:10.195196 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.195340 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.195482 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":69,"threads_started":1}
I20260928 12:59:10.198668 21598 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:59:10.199461 21598 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:59:10.204736 21598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: Generated new TSK 0
I20260928 12:59:10.204947 21598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:59:10.235203 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:10.319342 21612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:10.319653 21612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:10.319685 21612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:10.320983 21612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:10.321034 21612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:10.321063 21612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:59:10.322582 21612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:10.322858 21612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:10.323110 21612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:10.328485 21618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.328477 21617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.328537 21612 server_base.cc:1034] running on GCE node
W20260928 12:59:10.328477 21620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.328883 21612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:10.329125 21612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:10.330266 21612 hybrid_clock.cc:648] HybridClock initialized: now 1790600350330248 us; error 33 us; skew 500 ppm
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:10.333001 21612 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:59:10.333474 21612 webserver.cc:460] Webserver started at http://127.11.11.65:42259/ using document root <none> and password file <none>
I20260928 12:59:10.333714 21612 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:10.333783 21612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:10.333937 21612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:10.335033 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "25eb7c585374424aa882ab1cd57523c2"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "5a865efc2afde0ea9205f4fb1ece975a"
server_key_iv: "e968e3706b3e95f29e2051b26f9dadce"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.335433 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "25eb7c585374424aa882ab1cd57523c2"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "5a865efc2afde0ea9205f4fb1ece975a"
server_key_iv: "e968e3706b3e95f29e2051b26f9dadce"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.336923 21612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:59:10.337733 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.337913 21612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:10.338012 21612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "25eb7c585374424aa882ab1cd57523c2"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "5a865efc2afde0ea9205f4fb1ece975a"
server_key_iv: "e968e3706b3e95f29e2051b26f9dadce"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.338096 21612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:10.359769 21612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:10.361959 21612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:10.362151 21612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:10.362411 21612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:10.362744 21612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:10.362802 21612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.362838 21612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:10.362881 21612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.368597 21612 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:43641
I20260928 12:59:10.368647 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:43641 every 8 connection(s)
I20260928 12:59:10.369026 21612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:59:10.370347 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21612
I20260928 12:59:10.370448 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:59:10.370639 11309 external_mini_cluster.cc:1428] Setting key 70ac74d600d7cac0b82fded134e4bd70
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 12:59:10.380488 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.369811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:46853 (local address 127.11.11.126:37837)
0928 12:59:10.369858 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.369860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.370123 (+ 263us) server_negotiation.cc:407] Connection header received
0928 12:59:10.370756 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.370759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.370776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.370855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:10.371988 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.372811 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.374327 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.374695 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.374772 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.377261 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:10.377262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:10.377263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:10.377272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:10.378974 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.379401 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.379402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.379403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.379470 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.379728 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.379729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.379729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.379881 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:10.379968 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.380341 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.380416 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:59:10.380679 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.369783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37837 (local address 127.11.11.65:46853)
0928 12:59:10.369962 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.370008 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.370605 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.370960 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.370966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.371674 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:10.371876 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.371880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.372929 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.372931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.374235 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.374238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.374788 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.374789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.374892 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.375475 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:10.375485 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:10.377163 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:10.379075 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.379082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.379084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.379318 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.379548 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.379548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.379548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.379660 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.380020 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:10.380026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:10.380200 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.380289 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.380421 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260928 12:59:10.381284 21741 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37837
I20260928 12:59:10.381389 21741 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:10.381552 21741 heartbeater.cc:511] Master 127.11.11.126:37837 requested a full tablet report, sending...
I20260928 12:59:10.382181 21550 ts_manager.cc:194] Registered new tserver with Master: 25eb7c585374424aa882ab1cd57523c2 (127.11.11.65:43641)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
I20260928 12:59:10.383026 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:46853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:59:10.399590 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:10.482309 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:10.482553 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:10.482594 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:10.483980 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:10.484086 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:10.484169 21746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:59:10.485670 21746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:10.485966 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:10.486227 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:10.491365 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.491377 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.491416 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.491576 21746 server_base.cc:1034] running on GCE node
I20260928 12:59:10.491783 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:10.492093 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:10.493279 21746 hybrid_clock.cc:648] HybridClock initialized: now 1790600350493272 us; error 36 us; skew 500 ppm
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:10.495863 21746 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:59:10.496392 21746 webserver.cc:460] Webserver started at http://127.11.11.66:36867/ using document root <none> and password file <none>
I20260928 12:59:10.496640 21746 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:10.496709 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:10.496857 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:10.497936 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "f0909cc7d2ea41a190bde6e17517f850"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "0984ce2d31b3d61c695eeb9e0e066ff0"
server_key_iv: "193c0f3a7205b2c57ca27f13e3796ce9"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.498322 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "f0909cc7d2ea41a190bde6e17517f850"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "0984ce2d31b3d61c695eeb9e0e066ff0"
server_key_iv: "193c0f3a7205b2c57ca27f13e3796ce9"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.499650 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:59:10.500456 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.500626 21746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.500733 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "f0909cc7d2ea41a190bde6e17517f850"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "0984ce2d31b3d61c695eeb9e0e066ff0"
server_key_iv: "193c0f3a7205b2c57ca27f13e3796ce9"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.500810 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:10.539409 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:10.541620 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:10.541785 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:10.542078 21746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:10.542469 21746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:10.542531 21746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.542573 21746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:10.542600 21746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.548851 21746 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:32957
I20260928 12:59:10.548938 21874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:32957 every 8 connection(s)
I20260928 12:59:10.549286 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:59:10.555105 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21746
I20260928 12:59:10.555189 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:59:10.555402 11309 external_mini_cluster.cc:1428] Setting key 23aee4071b99fc364374c1b4242c45da
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:10.559587 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.550062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:58937 (local address 127.11.11.126:37837)
0928 12:59:10.550148 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.550150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.550315 (+ 165us) server_negotiation.cc:407] Connection header received
0928 12:59:10.550790 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.550792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.550802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.550855 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:10.551785 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.552369 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.553379 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.553639 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.553711 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.556477 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:10.556478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:10.556478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:10.556484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:10.558108 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.558519 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.558520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.558521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.558588 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.558839 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.558840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.558840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.558997 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:10.559071 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.559343 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.559444 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:59:10.559608 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.550015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37837 (local address 127.11.11.66:58937)
0928 12:59:10.550182 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.550220 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.550704 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.550912 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.550916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.551494 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:10.551706 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.551710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.552448 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.552449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.553287 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.553290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.553720 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.553722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.553893 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.554527 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:10.554537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:10.556381 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:10.558196 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.558202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.558204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.558435 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.558669 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.558669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.558669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.558773 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.559068 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:10.559073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:10.559207 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.559297 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.559432 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260928 12:59:10.560482 21875 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37837
I20260928 12:59:10.560608 21875 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:10.560842 21875 heartbeater.cc:511] Master 127.11.11.126:37837 requested a full tablet report, sending...
I20260928 12:59:10.561251 21550 ts_manager.cc:194] Registered new tserver with Master: f0909cc7d2ea41a190bde6e17517f850 (127.11.11.66:32957)
I20260928 12:59:10.561659 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:58937
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:59:10.582453 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:10.665637 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:10.665870 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:10.665896 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:10.667049 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:10.667094 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:10.667129 21880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:59:10.668629 21880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:10.668844 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:10.669032 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:10.674376 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.674446 21880 server_base.cc:1034] running on GCE node
W20260928 12:59:10.674383 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:10.674386 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:10.674851 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:10.675149 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:10.676322 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790600350676318 us; error 39 us; skew 500 ppm
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:10.678992 21880 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:59:10.679415 21880 webserver.cc:460] Webserver started at http://127.11.11.67:37771/ using document root <none> and password file <none>
I20260928 12:59:10.679603 21880 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:10.679647 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:10.679740 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:59:10.680850 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "50f657d61bae4552b91b1addcbbce8c8"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "bbf46a2951be5d8d6d90c0b3dfbf23b1"
server_key_iv: "7e80ea8733d8e4b8fd9e68c77f11a1e0"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.681180 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "50f657d61bae4552b91b1addcbbce8c8"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "bbf46a2951be5d8d6d90c0b3dfbf23b1"
server_key_iv: "7e80ea8733d8e4b8fd9e68c77f11a1e0"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.682732 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:59:10.683583 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.683769 21880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:59:10.683879 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "50f657d61bae4552b91b1addcbbce8c8"
format_stamp: "Formatted at 2026-09-28 12:59:10 on dist-test-slave-8sjz"
server_key: "bbf46a2951be5d8d6d90c0b3dfbf23b1"
server_key_iv: "7e80ea8733d8e4b8fd9e68c77f11a1e0"
server_key_version: "encryptionkey@0"
I20260928 12:59:10.683962 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:10.691530 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:10.693539 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:10.693718 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:10.693974 21880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:10.694306 21880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:10.694362 21880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.694406 21880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:10.694434 21880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:10.700589 21880 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:44963
I20260928 12:59:10.700639 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:44963 every 8 connection(s)
I20260928 12:59:10.701004 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:59:10.707557 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 21880
I20260928 12:59:10.707646 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:59:10.707865 11309 external_mini_cluster.cc:1428] Setting key 91de40037b9477a747baea99f595099b
Sep 28 12:59:10 dist-test-slave-8sjz krb5kdc[21505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600350, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:10.711596 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.701826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:32961 (local address 127.11.11.126:37837)
0928 12:59:10.701952 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.701955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.702183 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:59:10.702766 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.702767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.702780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.702844 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:10.704194 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.704815 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.706118 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.706309 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.706378 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.709152 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:10.709153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:10.709154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:10.709160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:10.710407 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.710816 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.710817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.710817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.710867 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.711078 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.711079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.711079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.711181 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:10.711231 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.711469 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.711531 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 12:59:10.711853 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.701809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:37837 (local address 127.11.11.67:32961)
0928 12:59:10.702017 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.702077 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.702648 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.702958 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.702962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.703747 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:10.703968 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.703974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.704955 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.704956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.706022 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.706024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.706398 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.706399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.706568 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.707161 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:10.707171 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:10.709055 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:10.710477 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.710483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.710485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.710749 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.710926 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:10.710927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:10.710927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:10.711026 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:10.711236 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:10.711240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:10.711360 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.711443 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.711603 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 12:59:10.712339 22009 heartbeater.cc:347] Connected to a master server at 127.11.11.126:37837
I20260928 12:59:10.712435 22009 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:10.712589 22009 heartbeater.cc:511] Master 127.11.11.126:37837 requested a full tablet report, sending...
I20260928 12:59:10.712953 21550 ts_manager.cc:194] Registered new tserver with Master: 50f657d61bae4552b91b1addcbbce8c8 (127.11.11.67:44963)
I20260928 12:59:10.713305 21550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:32961
I20260928 12:59:10.719628 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:59:10.726965 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.721022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.11.11.126:37837)
0928 12:59:10.721084 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.721086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.721216 (+ 130us) server_negotiation.cc:407] Connection header received
0928 12:59:10.721255 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.721256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.721265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.721316 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:10.721893 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.722376 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.723091 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.723279 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.723336 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.724687 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:10.724688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:10.724688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:10.724694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:10.725958 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.726274 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.726275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.726275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.726326 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:10.726487 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:10.726488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:10.726488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:10.726644 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:10.726720 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.726834 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.726896 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:59:10.728636 21550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:59:10.730286 21550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:59:10.737166 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.732773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32957 (local address 127.0.0.1:34822)
0928 12:59:10.733098 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.733154 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.733231 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.733562 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.733563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.733565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:10.733761 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.733765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.734766 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.734767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.736058 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.736061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.736818 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.736819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.736901 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.736915 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.737020 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.737080 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260928 12:59:10.737177 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.732874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.11.11.66:32957)
0928 12:59:10.733145 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.733148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.733331 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:59:10.733381 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.733384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.733401 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.733471 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:10.733852 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.734668 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.736165 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.736712 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.736815 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.736885 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.737039 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.737080 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260928 12:59:10.737412 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.732773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:44963 (local address 127.0.0.1:54878)
0928 12:59:10.732950 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.733261 (+ 311us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.733310 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.733614 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.733615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.733618 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:10.733717 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.733719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.735097 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.735098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.736171 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.736173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.737169 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.737170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.737232 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.737236 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.737285 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.737336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260928 12:59:10.737596 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.732960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.11.11.67:44963)
0928 12:59:10.733129 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.733135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.733375 (+ 240us) server_negotiation.cc:407] Connection header received
0928 12:59:10.733418 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.733422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.733438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.733583 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:10.733808 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.734988 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.736258 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.736990 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.737083 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.737136 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.737484 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.737526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260928 12:59:10.738519 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.732979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43641 (local address 127.0.0.1:44644)
0928 12:59:10.733188 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:10.733202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:10.733264 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:10.733456 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:10.733460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:10.733462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:10.733717 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.733720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.736131 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.736132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.737277 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:10.737279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.738270 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.738271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:10.738335 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.738339 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:59:10.738403 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:59:10.738444 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260928 12:59:10.738389 21809 tablet_service.cc:1467] Processing CreateTablet for tablet db4c37ed5a564f95acda7992fb25c738 (DEFAULT_TABLE table=test-table [id=2165aa2e861543d384225b41a0f7adbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:10.738545 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:10.733092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.11.11.65:43641)
0928 12:59:10.733265 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:10.733269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:10.733279 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:59:10.733328 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:10.733330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:10.733346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:10.733445 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:10.733806 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.734987 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.737366 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:10.738183 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:10.738260 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:10.738311 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:10.738431 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:59:10.738473 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260928 12:59:10.738560 21943 tablet_service.cc:1467] Processing CreateTablet for tablet db4c37ed5a564f95acda7992fb25c738 (DEFAULT_TABLE table=test-table [id=2165aa2e861543d384225b41a0f7adbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:10.738792 21809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4c37ed5a564f95acda7992fb25c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:10.738914 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4c37ed5a564f95acda7992fb25c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:10.741011 21675 tablet_service.cc:1467] Processing CreateTablet for tablet db4c37ed5a564f95acda7992fb25c738 (DEFAULT_TABLE table=test-table [id=2165aa2e861543d384225b41a0f7adbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:10.741369 21675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4c37ed5a564f95acda7992fb25c738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:10.741953 22026 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Bootstrap starting.
I20260928 12:59:10.742650 22026 tablet_bootstrap.cc:654] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:10.743141 22026 log.cc:824] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:10.743893 22026 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: No bootstrap required, opened a new log
I20260928 12:59:10.743957 22026 ts_tablet_manager.cc:1397] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:59:10.744521 22029 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Bootstrap starting.
I20260928 12:59:10.744725 22027 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Bootstrap starting.
I20260928 12:59:10.745210 22029 tablet_bootstrap.cc:654] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:10.745395 22027 tablet_bootstrap.cc:654] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:10.745805 22029 log.cc:824] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:10.745954 22027 log.cc:824] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:10.746171 22026 raft_consensus.cc:348] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.746367 22026 raft_consensus.cc:374] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:10.746419 22026 raft_consensus.cc:729] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0909cc7d2ea41a190bde6e17517f850, State: Initialized, Role: FOLLOWER
I20260928 12:59:10.746541 22026 consensus_queue.cc:260] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.746773 22029 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: No bootstrap required, opened a new log
I20260928 12:59:10.746840 22026 ts_tablet_manager.cc:1428] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:59:10.746850 22029 ts_tablet_manager.cc:1397] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:59:10.746878 22027 tablet_bootstrap.cc:492] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: No bootstrap required, opened a new log
I20260928 12:59:10.746948 22027 ts_tablet_manager.cc:1397] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:59:10.748183 21875 heartbeater.cc:503] Master 127.11.11.126:37837 was elected leader, sending a full tablet report...
I20260928 12:59:10.748422 22029 raft_consensus.cc:348] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.748595 22029 raft_consensus.cc:374] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:10.748667 22029 raft_consensus.cc:729] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eb7c585374424aa882ab1cd57523c2, State: Initialized, Role: FOLLOWER
I20260928 12:59:10.748795 22029 consensus_queue.cc:260] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.749101 22029 ts_tablet_manager.cc:1428] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:59:10.749121 21741 heartbeater.cc:503] Master 127.11.11.126:37837 was elected leader, sending a full tablet report...
I20260928 12:59:10.749234 22027 raft_consensus.cc:348] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.749393 22027 raft_consensus.cc:374] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:10.749445 22027 raft_consensus.cc:729] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f657d61bae4552b91b1addcbbce8c8, State: Initialized, Role: FOLLOWER
I20260928 12:59:10.749567 22027 consensus_queue.cc:260] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:10.749864 22027 ts_tablet_manager.cc:1428] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:59:10.749889 22009 heartbeater.cc:503] Master 127.11.11.126:37837 was elected leader, sending a full tablet report...
W20260928 12:59:10.799940 21876 tablet.cc:2369] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:59:10.869807 21742 tablet.cc:2369] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:59:10.951601 22010 tablet.cc:2369] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:59:11.007973 22034 raft_consensus.cc:482] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:59:11.008143 22034 raft_consensus.cc:504] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:11.008563 22034 leader_election.cc:288] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0909cc7d2ea41a190bde6e17517f850 (127.11.11.66:32957), 25eb7c585374424aa882ab1cd57523c2 (127.11.11.65:43641)
I20260928 12:59:11.011538 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.008871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:38629 (local address 127.11.11.66:32957)
0928 12:59:11.008948 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:11.008950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:11.008964 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:59:11.009008 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:11.009009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:11.009019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:11.009092 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:11.009268 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.009825 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.010790 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.011179 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.011267 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:11.011275 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:11.011434 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 12:59:11.011471 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:59:11.011545 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.008793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32957 (local address 127.11.11.67:38629)
0928 12:59:11.008868 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:11.008883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:11.008950 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:11.009108 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:11.009109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:11.009111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:11.009201 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:11.009204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.009914 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.009915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:11.010686 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:11.010688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.011265 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.011266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:11.011351 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:11.011359 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 12:59:11.011428 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:59:11.011471 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 12:59:11.011539 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.009182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:45385 (local address 127.11.11.65:43641)
0928 12:59:11.009239 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:11.009241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:11.009248 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:59:11.009278 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:11.009279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:11.009286 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:11.009331 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:11.009453 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.009896 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.010813 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.011172 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.011263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:11.011272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:11.011428 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 12:59:11.011467 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:59:11.011554 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.008935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43641 (local address 127.11.11.67:45385)
0928 12:59:11.009131 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:11.009140 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:11.009183 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:11.009334 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:11.009335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:11.009336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:11.009394 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:11.009395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.009979 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.009981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:11.010714 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:11.010716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.011261 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.011262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:11.011334 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:11.011344 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 12:59:11.011419 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:59:11.011462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260928 12:59:11.011976 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4c37ed5a564f95acda7992fb25c738" candidate_uuid: "50f657d61bae4552b91b1addcbbce8c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7c585374424aa882ab1cd57523c2" is_pre_election: true
I20260928 12:59:11.012002 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4c37ed5a564f95acda7992fb25c738" candidate_uuid: "50f657d61bae4552b91b1addcbbce8c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0909cc7d2ea41a190bde6e17517f850" is_pre_election: true
I20260928 12:59:11.012193 21695 raft_consensus.cc:2454] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f657d61bae4552b91b1addcbbce8c8 in term 0.
I20260928 12:59:11.012214 21829 raft_consensus.cc:2454] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50f657d61bae4552b91b1addcbbce8c8 in term 0.
I20260928 12:59:11.012439 21897 leader_election.cc:302] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f657d61bae4552b91b1addcbbce8c8, f0909cc7d2ea41a190bde6e17517f850; no voters:
I20260928 12:59:11.012588 22034 raft_consensus.cc:2781] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:59:11.012650 22034 raft_consensus.cc:482] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:59:11.012674 22034 raft_consensus.cc:3037] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:11.013368 22034 raft_consensus.cc:504] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:11.013506 22034 leader_election.cc:288] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [CANDIDATE]: Term 1 election: Requested vote from peers f0909cc7d2ea41a190bde6e17517f850 (127.11.11.66:32957), 25eb7c585374424aa882ab1cd57523c2 (127.11.11.65:43641)
I20260928 12:59:11.013671 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4c37ed5a564f95acda7992fb25c738" candidate_uuid: "50f657d61bae4552b91b1addcbbce8c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0909cc7d2ea41a190bde6e17517f850"
I20260928 12:59:11.013701 21695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4c37ed5a564f95acda7992fb25c738" candidate_uuid: "50f657d61bae4552b91b1addcbbce8c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eb7c585374424aa882ab1cd57523c2"
I20260928 12:59:11.013748 21829 raft_consensus.cc:3037] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:11.013803 21695 raft_consensus.cc:3037] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:11.014478 21829 raft_consensus.cc:2454] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f657d61bae4552b91b1addcbbce8c8 in term 1.
I20260928 12:59:11.014549 21695 raft_consensus.cc:2454] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50f657d61bae4552b91b1addcbbce8c8 in term 1.
I20260928 12:59:11.014654 21897 leader_election.cc:302] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50f657d61bae4552b91b1addcbbce8c8, f0909cc7d2ea41a190bde6e17517f850; no voters:
I20260928 12:59:11.014758 22034 raft_consensus.cc:2781] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:11.014921 22034 raft_consensus.cc:686] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 1 LEADER]: Becoming Leader. State: Replica: 50f657d61bae4552b91b1addcbbce8c8, State: Running, Role: LEADER
I20260928 12:59:11.015026 22034 consensus_queue.cc:237] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } }
I20260928 12:59:11.015973 21549 catalog_manager.cc:5495] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50f657d61bae4552b91b1addcbbce8c8 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "50f657d61bae4552b91b1addcbbce8c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50f657d61bae4552b91b1addcbbce8c8" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 44963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 } health_report { overall_health: UNKNOWN } } }
I20260928 12:59:11.023757 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.021240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.11.11.67:44963)
0928 12:59:11.021321 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:11.021323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:11.021330 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:59:11.021362 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:11.021363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:11.021379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:11.021441 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:59:11.021588 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.022105 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.022875 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:11.023105 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:11.023176 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:11.023304 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:59:11.023419 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:59:11.023542 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:11.023558 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 12:59:11.023614 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:59:11.026213 21695 raft_consensus.cc:1261] T db4c37ed5a564f95acda7992fb25c738 P 25eb7c585374424aa882ab1cd57523c2 [term 1 FOLLOWER]: Refusing update from remote peer 50f657d61bae4552b91b1addcbbce8c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:59:11.026254 21829 raft_consensus.cc:1261] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 1 FOLLOWER]: Refusing update from remote peer 50f657d61bae4552b91b1addcbbce8c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:59:11.026629 22034 consensus_queue.cc:1035] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eb7c585374424aa882ab1cd57523c2" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:59:11.026804 22036 consensus_queue.cc:1035] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0909cc7d2ea41a190bde6e17517f850" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:59:11.029965 22039 mvcc.cc:205] Tried to move back new op lower bound from 7334299037799981056 to 7334299037758337024. Current Snapshot: MvccSnapshot[applied={T|T < 7334299037799981056}]
I20260928 12:59:11.033345 21549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:59:11.033476 21549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:59:11.034272 21549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b94eeab2eb44e6c89dd2e9e9f42bf8a: Sending DeleteTablet for 3 replicas of tablet db4c37ed5a564f95acda7992fb25c738
I20260928 12:59:11.034627 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet db4c37ed5a564f95acda7992fb25c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:11 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:54878
I20260928 12:59:11.034680 21675 tablet_service.cc:1514] Processing DeleteTablet for tablet db4c37ed5a564f95acda7992fb25c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:11 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:44644
I20260928 12:59:11.034767 21809 tablet_service.cc:1514] Processing DeleteTablet for tablet db4c37ed5a564f95acda7992fb25c738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:11 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:34822
I20260928 12:59:11.034979 22050 tablet_replica.cc:330] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: stopping tablet replica
I20260928 12:59:11.034969 22048 tablet_replica.cc:330] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: stopping tablet replica
I20260928 12:59:11.035095 22050 raft_consensus.cc:2229] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:59:11.035132 22048 raft_consensus.cc:2229] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:59:11.035228 22050 raft_consensus.cc:2258] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:59:11.035274 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21612
I20260928 12:59:11.035419 22048 raft_consensus.cc:2258] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:59:11.036321 22050 ts_tablet_manager.cc:1905] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:59:11.037223 22048 ts_tablet_manager.cc:1905] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:59:11.038404 22050 ts_tablet_manager.cc:1918] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:59:11.038491 22050 log.cc:1196] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/db4c37ed5a564f95acda7992fb25c738
I20260928 12:59:11.038762 22050 ts_tablet_manager.cc:1939] T db4c37ed5a564f95acda7992fb25c738 P f0909cc7d2ea41a190bde6e17517f850: Deleting consensus metadata
I20260928 12:59:11.039314 21535 catalog_manager.cc:4841] TS f0909cc7d2ea41a190bde6e17517f850 (127.11.11.66:32957): tablet db4c37ed5a564f95acda7992fb25c738 (table test-table [id=2165aa2e861543d384225b41a0f7adbb]) successfully deleted
I20260928 12:59:11.039639 22048 ts_tablet_manager.cc:1918] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:59:11.039736 22048 log.cc:1196] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/db4c37ed5a564f95acda7992fb25c738
I20260928 12:59:11.040078 22048 ts_tablet_manager.cc:1939] T db4c37ed5a564f95acda7992fb25c738 P 50f657d61bae4552b91b1addcbbce8c8: Deleting consensus metadata
I20260928 12:59:11.040560 21536 catalog_manager.cc:4841] TS 50f657d61bae4552b91b1addcbbce8c8 (127.11.11.67:44963): tablet db4c37ed5a564f95acda7992fb25c738 (table test-table [id=2165aa2e861543d384225b41a0f7adbb]) successfully deleted
W20260928 12:59:11.043615 21535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:43641 (error 108)
I20260928 12:59:11.043861 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21746
I20260928 12:59:11.044832 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:11.044657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43641 (local address 127.0.0.1:44650)
0928 12:59:11.044732 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:11.044785 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:43641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 12:59:11.045038 21535 catalog_manager.cc:4584] TS 25eb7c585374424aa882ab1cd57523c2 (127.11.11.65:43641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db4c37ed5a564f95acda7992fb25c738: Network error: Client connection negotiation failed: client connection to 127.11.11.65:43641: connect: Connection refused (error 111)
I20260928 12:59:11.050707 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21880
I20260928 12:59:11.056193 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 21518
2026-09-28T12:59:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2173 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:59:11 dist-test-slave-8sjz krb5kdc[22054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:59:11 dist-test-slave-8sjz krb5kdc[22054](info): set up 2 sockets
Sep 28 12:59:11 dist-test-slave-8sjz krb5kdc[22054](info): commencing operation
krb5kdc: starting...
W20260928 12:59:12.122423 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 12:59:12 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:59:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangeradmin_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangeradmin_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangerlookup_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangerlookup_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
I20260928 12:59:12.295054 11309 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNYzOw4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/postgres -l logfile start
2026-09-28 12:59:14.912 UTC [22084] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:59:14.913 UTC [22084] LOG: listening on IPv4 address "127.11.11.84", port 35419
2026-09-28 12:59:14.917 UTC [22084] LOG: listening on Unix socket "/tmp/.s.PGSQL.35419"
2026-09-28 12:59:14.947 UTC [22087] LOG: database system was shut down at 2026-09-28 12:59:13 UTC
2026-09-28 12:59:14.950 UTC [22084] LOG: database system is ready to accept connections
I20260928 12:59:15.902119 11309 mini_postgres.cc:96] Postgres bound to 35419
2026-09-28 12:59:15.908 UTC [22095] FATAL: database "slave" does not exist
127.11.11.84:35419 - accepting connections
I20260928 12:59:15.909482 11309 mini_ranger.cc:161] Starting Ranger...
I20260928 12:59:15.925014 11309 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:59:16.439126 11309 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:59:16.439194 11309 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin
2026-09-28 12:59:16,742 [I] DB FLAVOR :POSTGRES
2026-09-28 12:59:16,742 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:59:16,742 [I] Checking connection..
2026-09-28 12:59:16,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:59:17,022 [I] Checking connection passed.
2026-09-28 12:59:17,022 [I] --------- Verifying version history table ---------
2026-09-28 12:59:17,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:59:17,276 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:59:17,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:59:17,537 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:59:17,537 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:59:17,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:59:17.769 UTC [22159] WARNING: there is no transaction in progress
2026-09-28 12:59:17,775 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:59:17,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:59:18,017 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:59:18,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:59:18,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:59:18,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:59:18,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8sjz.c.gcp-upstream.internal','N') ;"
2026-09-28 12:59:18,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:59:18,976 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:59:18,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:59:19,244 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:59:19,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:59:19,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:59:19,734 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:59:19,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:59:20.719 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.737 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.753 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.769 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.788 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.805 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.861 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.868 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.882 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.892 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.903 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.917 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.927 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.935 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.943 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:20.950 UTC [22267] WARNING: there is no transaction in progress
2026-09-28 12:59:21,691 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:59:21,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:59:21,929 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:59:21,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:59:22,182 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:59:22,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:59:22,437 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:59:22,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:59:22,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:59:22,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8sjz.c.gcp-upstream.internal';"
2026-09-28 12:59:23,132 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:59:23,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:59:23,368 [I] DB_PATCHES have already been applied
I20260928 12:59:23.374413 11309 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNYzOw4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:59:23.374514 11309 mini_ranger.cc:193] Using host: 127.11.11.84
I20260928 12:59:23.377777 11309 mini_ranger.cc:227] Ranger admin URL: http://127.11.11.84:53373
Sep 28, 2026 12:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:59:24.511493 11309 mini_ranger.cc:160] Time spent starting Ranger: real 8.602s user 0.000s sys 0.009s
Sep 28, 2026 12:59:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:59:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 12:59:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:59:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:59:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:59:46.667503 11309 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangerkms_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/rangerkms_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.11.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.11.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:59:46 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600386, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:59:46.758616 11309 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:59:46.768548 11309 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:59:47.771325 11309 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:59:47.771430 11309 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms
I20260928 12:59:47.771450 11309 mini_ranger_kms.cc:227] Using postgres at 127.11.11.84:35419
2026-09-28 12:59:48,120 [I] DB FLAVOR :POSTGRES
2026-09-28 12:59:48,120 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:59:48,120 [I] Checking connection
2026-09-28 12:59:48,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:59:48,408 [I] connection success
2026-09-28 12:59:48,408 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:59:48,408 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:59:48,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:59:48,657 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:59:48,657 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:59:48,657 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:59:48,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:35419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:59:48,925 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:59:49.449667 11309 mini_ranger_kms.cc:310] Created kms service
I20260928 12:59:49.677093 11309 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:59:49.722452 11309 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:59:49 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600386, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:50.166069 11309 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:59:50.166198 11309 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:59:50.166220 11309 mini_ranger_kms.cc:240] Using host: 127.11.11.84
I20260928 12:59:50.169860 11309 mini_ranger_kms.cc:276] Ranger KMS PID: 22447
I20260928 12:59:50.169992 11309 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.11.84:55329
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:59:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:59:52.425916 11309 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.667s user 0.000s sys 0.330s
I20260928 12:59:52.426033 11309 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:59:52.426118 11309 mini_ranger_kms.cc:401] 127.11.11.84:55329/kms/v1/keys
Sep 28 12:59:52 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790600392, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:59:53 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600392, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
Sep 28 12:59:53 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600392, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
Sep 28 12:59:56 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 12:59:56.072930 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38049
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:38049
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:55329/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:56.186810 22498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:56.187075 22498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:56.187119 22498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:56.188464 22498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:59:56.188534 22498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:56.188558 22498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:56.188591 22498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:59:56.188616 22498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:59:56.190321 22498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.11.84:55329/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:38049
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:56.190609 22498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:56.190840 22498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:56.196161 22506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:56.196146 22503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:56.196151 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:56.196390 22498 server_base.cc:1034] running on GCE node
I20260928 12:59:56.196714 22498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:56.197065 22498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:56.198249 22498 hybrid_clock.cc:648] HybridClock initialized: now 1790600396198222 us; error 40 us; skew 500 ppm
Sep 28 12:59:56 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:56.201253 22498 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 12:59:56.201779 22498 webserver.cc:460] Webserver started at http://127.11.11.126:36025/ using document root <none> and password file <none>
I20260928 12:59:56.202029 22498 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:56.202091 22498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:56.202243 22498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:59:56 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600396, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:56.265945 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "0b17f330abc745ba957dc56e61e32750"
format_stamp: "Formatted at 2026-09-28 12:59:56 on dist-test-slave-8sjz"
server_key: "9ab4e073a3559febcbf47ce5aec8311d"
server_key_iv: "0d3162773bbf479151d3f8ad937040bb"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:56.266435 22498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "0b17f330abc745ba957dc56e61e32750"
format_stamp: "Formatted at 2026-09-28 12:59:56 on dist-test-slave-8sjz"
server_key: "9ab4e073a3559febcbf47ce5aec8311d"
server_key_iv: "0d3162773bbf479151d3f8ad937040bb"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:56.267946 22498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:59:56.282720 22513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:56.282989 22498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:56.283103 22498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "0b17f330abc745ba957dc56e61e32750"
format_stamp: "Formatted at 2026-09-28 12:59:56 on dist-test-slave-8sjz"
server_key: "9ab4e073a3559febcbf47ce5aec8311d"
server_key_iv: "0d3162773bbf479151d3f8ad937040bb"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:56.283205 22498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:56.301244 22498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:56.303437 22498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:56.303602 22498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:56.308707 22498 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:38049
I20260928 12:59:56.308746 22565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:38049 every 8 connection(s)
I20260928 12:59:56.309147 22498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 12:59:56.309863 22566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:56.310570 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 22498
I20260928 12:59:56.310683 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 12:59:56.310889 11309 external_mini_cluster.cc:1428] Setting key b09eca59897fb5c1e1de56cf84e21b37
I20260928 12:59:56.313344 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: Bootstrap starting.
I20260928 12:59:56.313988 22566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:56.314288 22566 log.cc:824] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:56.315006 22566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: No bootstrap required, opened a new log
I20260928 12:59:56.317518 22566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } }
I20260928 12:59:56.317724 22566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:56.317763 22566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b17f330abc745ba957dc56e61e32750, State: Initialized, Role: FOLLOWER
I20260928 12:59:56.317986 22566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } }
I20260928 12:59:56.318126 22566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:59:56.318190 22566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:59:56.318260 22566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:56.319011 22566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } }
I20260928 12:59:56.319140 22566 leader_election.cc:302] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b17f330abc745ba957dc56e61e32750; no voters:
Sep 28 12:59:56 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:56.319357 22566 leader_election.cc:288] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:59:56.319660 22566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:59:56.319895 22571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:56.320132 22571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [term 1 LEADER]: Becoming Leader. State: Replica: 0b17f330abc745ba957dc56e61e32750, State: Running, Role: LEADER
I20260928 12:59:56.320322 22571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } }
I20260928 12:59:56.320750 22571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b17f330abc745ba957dc56e61e32750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } } }
I20260928 12:59:56.320917 22571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:56.321089 22571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b17f330abc745ba957dc56e61e32750. Latest consensus state: current_term: 1 leader_uuid: "0b17f330abc745ba957dc56e61e32750" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b17f330abc745ba957dc56e61e32750" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38049 } } }
I20260928 12:59:56.321219 22571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750 [sys.catalog]: This master's current role is: LEADER
I20260928 12:59:56.321166 22566 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:59:56.321554 22585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:59:56.322129 22585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:59:56.332993 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:56.311499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.11.11.126:38049)
0928 12:59:56.311708 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:56.311718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:56.311753 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:59:56.312400 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:56.312410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:56.312511 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:56.312679 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:56.313542 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:56.314963 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:56.316066 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:56.316745 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:56.316872 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:56.319884 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:56.319888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:56.319889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:56.319903 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:56.322032 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:56.323870 (+ 1838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:56.324001 (+ 131us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:56.324001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:56.325039 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:56.325427 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:56.325428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:56.325428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:56.326860 (+ 1432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:56.332284 (+ 5424us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:56.332439 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:59:56.332682 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260928 12:59:56.337455 22585 catalog_manager.cc:1347] Generated new cluster ID: 506eb76296ad467ab472ecc55322bfee
I20260928 12:59:56.337523 22585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:59:56.370009 22585 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:59:56.370855 22585 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:59:56.381088 22585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: Generated new TSK 0
I20260928 12:59:56.381403 22585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 12:59:57.314692 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:55329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:57.448318 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:57.448642 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:57.448680 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:57.450541 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:57.450613 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:57.450654 22611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 12:59:57.453254 22611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.11.84:55329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:57.453560 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:57.453882 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:57.462708 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:57.462813 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:57.463011 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:57.463743 22611 server_base.cc:1034] running on GCE node
I20260928 12:59:57.464126 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:57.464479 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:57.465669 22611 hybrid_clock.cc:648] HybridClock initialized: now 1790600397465649 us; error 35 us; skew 500 ppm
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:57.470201 22611 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 12:59:57.470824 22611 webserver.cc:460] Webserver started at http://127.11.11.65:34527/ using document root <none> and password file <none>
I20260928 12:59:57.471174 22611 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:57.471292 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:57.471470 22611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:57.487022 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "a3a84fd1ba7a48a28af83c29118d9aef"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce0ef594aedab3c2fab8c11c996656ec"
server_key_iv: "87ba46d3766f51c9619c82fe967961b9"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.487521 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "a3a84fd1ba7a48a28af83c29118d9aef"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce0ef594aedab3c2fab8c11c996656ec"
server_key_iv: "87ba46d3766f51c9619c82fe967961b9"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.489638 22611 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 12:59:57.506886 22626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.507164 22611 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260928 12:59:57.507261 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "a3a84fd1ba7a48a28af83c29118d9aef"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce0ef594aedab3c2fab8c11c996656ec"
server_key_iv: "87ba46d3766f51c9619c82fe967961b9"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.507382 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:57.524854 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:57.527890 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:57.528087 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:57.528600 22611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:57.529143 22611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:57.529260 22611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.529387 22611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:57.529471 22611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.541328 22611 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:43089
I20260928 12:59:57.541857 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 12:59:57.543388 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 22611
I20260928 12:59:57.543495 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 12:59:57.543686 11309 external_mini_cluster.cc:1428] Setting key e424dfbe84f099e8d092eb36b34c7cc6
I20260928 12:59:57.545245 22739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:43089 every 8 connection(s)
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:57.567839 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:57.545969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:54515 (local address 127.11.11.126:38049)
0928 12:59:57.547107 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:57.547110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:57.547326 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:59:57.548179 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:57.548180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:57.548196 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:57.548238 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:57.555799 (+ 7561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.556856 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.559174 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.559514 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.559583 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:57.563765 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:57.563767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:57.563768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:57.563778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:57.566012 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:57.566647 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:57.566649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:57.566649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:57.566719 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:57.567014 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:57.567015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:57.567016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:57.567173 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:57.567239 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:57.567630 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 12:59:57.567719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":68,"threads_started":1}
I20260928 12:59:57.575441 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:57.545967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38049 (local address 127.11.11.65:54515)
0928 12:59:57.547181 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:57.547235 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:57.548017 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:57.549321 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:57.549325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:57.549924 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:57.555666 (+ 5742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:57.555677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.557523 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.557525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:57.559045 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:57.559050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.559691 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.559692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:57.559911 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:57.560751 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:57.560762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:57.563622 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:57.566139 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:57.566149 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:57.566152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:57.566544 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:57.566789 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:57.566790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:57.566791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:57.566940 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:57.567276 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:57.567281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:57.567472 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 12:59:57.567594 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 12:59:57.567761 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":80,"threads_started":1}
I20260928 12:59:57.576453 22740 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38049
I20260928 12:59:57.576602 22740 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:57.576866 22740 heartbeater.cc:511] Master 127.11.11.126:38049 requested a full tablet report, sending...
I20260928 12:59:57.577557 22530 ts_manager.cc:194] Registered new tserver with Master: a3a84fd1ba7a48a28af83c29118d9aef (127.11.11.65:43089)
I20260928 12:59:57.578610 22530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:54515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 12:59:57.587082 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:55329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:57.725688 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:57.726008 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:57.726045 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:57.728070 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:57.728147 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:57.728192 22748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 12:59:57.730984 22748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.11.84:55329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:57.731307 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:57.731642 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:57.740311 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:57.740715 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:57.740746 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:57.741302 22748 server_base.cc:1034] running on GCE node
I20260928 12:59:57.741556 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:57.741848 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:57.742985 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790600397742962 us; error 34 us; skew 500 ppm
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:57.746867 22748 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 12:59:57.747541 22748 webserver.cc:460] Webserver started at http://127.11.11.66:42139/ using document root <none> and password file <none>
I20260928 12:59:57.747829 22748 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:57.747929 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:57.748126 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:57.762109 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "16874dc1a160468494e3fdd64b65e1ff"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce911d36c7f3c05582c6045c2d7527d8"
server_key_iv: "9f8aa444bdae747f975688f4a339fe9d"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.762795 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "16874dc1a160468494e3fdd64b65e1ff"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce911d36c7f3c05582c6045c2d7527d8"
server_key_iv: "9f8aa444bdae747f975688f4a339fe9d"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.766520 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 12:59:57.783408 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.784013 22748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:59:57.784166 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "16874dc1a160468494e3fdd64b65e1ff"
format_stamp: "Formatted at 2026-09-28 12:59:57 on dist-test-slave-8sjz"
server_key: "ce911d36c7f3c05582c6045c2d7527d8"
server_key_iv: "9f8aa444bdae747f975688f4a339fe9d"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:57.784274 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:57.799038 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:57.809608 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:57.813005 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:57.813355 22748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:57.813823 22748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:57.813875 22748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.813910 22748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:57.813926 22748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:57.838775 22748 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:43005
I20260928 12:59:57.839383 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 12:59:57.842826 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:43005 every 8 connection(s)
I20260928 12:59:57.844596 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 22748
I20260928 12:59:57.844710 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 12:59:57.844878 11309 external_mini_cluster.cc:1428] Setting key e4bb371cedd9ea7fa8ec2e76075f0df2
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Sep 28 12:59:57 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:57.865842 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:57.843426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:58541 (local address 127.11.11.126:38049)
0928 12:59:57.843502 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:57.843504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:57.843550 (+ 46us) server_negotiation.cc:407] Connection header received
0928 12:59:57.844711 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:57.844713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:57.844729 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:57.844761 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:57.845978 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.846843 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.848553 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.848831 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.848880 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:57.861446 (+ 12566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:57.861449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:57.861449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:57.861460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:57.863585 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:57.864327 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:57.864329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:57.864329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:57.864437 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:57.864845 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:57.864847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:57.864847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:57.865033 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:57.865111 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:57.865592 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 12:59:57.865716 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:59:57.866918 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:57.842972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38049 (local address 127.11.11.66:58541)
0928 12:59:57.843290 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:57.843354 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:57.844559 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:57.844956 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:57.844961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:57.845439 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:57.845861 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:57.845869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.846959 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.846959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:57.848446 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:57.848450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:57.849005 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:57.849006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:57.849213 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:57.858479 (+ 9266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:57.858504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:57.861226 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:57.863726 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:57.863739 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:57.863742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:57.864159 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:57.864540 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:57.864541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:57.864541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:57.864746 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:57.865153 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:57.865162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:57.865390 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 12:59:57.866285 (+ 895us) client_negotiation.cc:240] Negotiation successful
0928 12:59:57.866530 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":201,"threads_started":1}
I20260928 12:59:57.867797 22879 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38049
I20260928 12:59:57.867910 22879 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:57.868197 22879 heartbeater.cc:511] Master 127.11.11.126:38049 requested a full tablet report, sending...
I20260928 12:59:57.868748 22530 ts_manager.cc:194] Registered new tserver with Master: 16874dc1a160468494e3fdd64b65e1ff (127.11.11.66:43005)
I20260928 12:59:57.869326 22530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:58541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 12:59:57.895998 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:55329/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:59:58.091176 22885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:59:58.091738 22885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:59:58.091871 22885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:59:58.094605 22885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:59:58.094777 22885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:59:58.094906 22885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 12:59:58.098121 22885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.11.84:55329/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 12:59:58.098814 22885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:59:58.099249 22885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:59:58.107307 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:59:58.107324 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:58.109021 22885 server_base.cc:1034] running on GCE node
W20260928 12:59:58.113018 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:59:58.113447 22885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:59:58.113804 22885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:59:58.121230 22885 hybrid_clock.cc:648] HybridClock initialized: now 1790600398121188 us; error 59 us; skew 500 ppm
Sep 28 12:59:58 dist-test-slave-8sjz krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600398, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:59:58.125748 22885 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 12:59:58.126351 22885 webserver.cc:460] Webserver started at http://127.11.11.67:33997/ using document root <none> and password file <none>
I20260928 12:59:58.126679 22885 fs_manager.cc:359] Metadata directory not provided
I20260928 12:59:58.126746 22885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:59:58.126879 22885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:59:58 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600398, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:58.150779 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "645fa5f358a64c2a9fe2ced589bf346c"
format_stamp: "Formatted at 2026-09-28 12:59:58 on dist-test-slave-8sjz"
server_key: "83c8aacff9ae33d8095c5734e2cbc76a"
server_key_iv: "a7072f462699f215bc737e2ab8599698"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:58.151782 22885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "645fa5f358a64c2a9fe2ced589bf346c"
format_stamp: "Formatted at 2026-09-28 12:59:58 on dist-test-slave-8sjz"
server_key: "83c8aacff9ae33d8095c5734e2cbc76a"
server_key_iv: "a7072f462699f215bc737e2ab8599698"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:58.159643 22885 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260928 12:59:58.172596 22900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:58.172847 22885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:59:58.172931 22885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "645fa5f358a64c2a9fe2ced589bf346c"
format_stamp: "Formatted at 2026-09-28 12:59:58 on dist-test-slave-8sjz"
server_key: "83c8aacff9ae33d8095c5734e2cbc76a"
server_key_iv: "a7072f462699f215bc737e2ab8599698"
server_key_version: "kuduclusterkey@0"
I20260928 12:59:58.173022 22885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:59:58.199415 22885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:59:58.210697 22885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:59:58.210999 22885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:59:58.211309 22885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:59:58.211812 22885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:59:58.211916 22885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:58.212003 22885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:59:58.212105 22885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:59:58.222903 22885 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:33139
I20260928 12:59:58.223502 22885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 12:59:58.229799 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 22885
I20260928 12:59:58.229930 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 12:59:58.230152 11309 external_mini_cluster.cc:1428] Setting key a9e280e5d38419f223767d1ec8e1ed40
I20260928 12:59:58.232453 23013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:33139 every 8 connection(s)
Sep 28 12:59:58 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600398, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 12:59:58.256474 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.241203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:59491 (local address 127.11.11.126:38049)
0928 12:59:58.241300 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.241302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.244396 (+ 3094us) server_negotiation.cc:407] Connection header received
0928 12:59:58.245193 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.245195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.245208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.245240 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:58.246200 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.246954 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.248722 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.248999 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.249045 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.251839 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:58.251841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:58.251841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:58.251849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:58.254562 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:58.255095 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:58.255096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:58.255097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:58.255159 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:58.255626 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:58.255627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:58.255627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:58.255787 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:58.255832 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.256273 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.256373 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 12:59:58.257332 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.240836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38049 (local address 127.11.11.67:59491)
0928 12:59:58.244214 (+ 3378us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.244293 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.245053 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.245458 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.245463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.245772 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:59:58.246084 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.246091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.247129 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.247130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.248620 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.248623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.249144 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.249145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.249291 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.249980 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:59:58.249992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:59:58.251726 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:59:58.254662 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:58.254670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:58.254673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:58.254999 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:58.255232 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:59:58.255232 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:59:58.255233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:59:58.255519 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:59:58.255882 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:59:58.255889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:59:58.256103 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.256887 (+ 784us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.257080 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3308,"thread_start_us":108,"threads_started":1}
I20260928 12:59:58.258193 23014 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38049
I20260928 12:59:58.258329 23014 heartbeater.cc:464] Registering TS with master...
I20260928 12:59:58.258533 23014 heartbeater.cc:511] Master 127.11.11.126:38049 requested a full tablet report, sending...
I20260928 12:59:58.258939 22530 ts_manager.cc:194] Registered new tserver with Master: 645fa5f358a64c2a9fe2ced589bf346c (127.11.11.67:33139)
I20260928 12:59:58.259389 22530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:59491
I20260928 12:59:58.264712 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:59:58.275362 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.266567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.11.11.126:38049)
0928 12:59:58.266614 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.266616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.267095 (+ 479us) server_negotiation.cc:407] Connection header received
0928 12:59:58.267198 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.267199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.267211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.267240 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:59:58.268207 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.269002 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.270130 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.270430 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.270471 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.272283 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:59:58.272284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:59:58.272285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:59:58.272292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:59:58.274065 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:58.274481 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:58.274482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:58.274483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:58.274552 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:59:58.274755 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:59:58.274756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:59:58.274756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:59:58.274886 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:59:58.274960 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.275176 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.275255 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:59:58.277940 22530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:59:58.282099 22530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:59:58.308867 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33139 (local address 127.0.0.1:50286)
0928 12:59:58.299710 (+ 14546us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.299724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.299785 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.300497 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.300497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.300499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:58.300527 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.300528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.303665 (+ 3137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.303666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.305023 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.305026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.308555 (+ 3529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.308556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.308650 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.308667 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.308738 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.308797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14460,"mutex_wait_us":4,"thread_start_us":152,"threads_started":1}
I20260928 12:59:58.312005 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43089 (local address 127.0.0.1:59436)
0928 12:59:58.286073 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.286120 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.286241 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.300325 (+ 14084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.300329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.300332 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:58.300432 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.300434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.305172 (+ 4738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.305173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.306304 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.306307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.311782 (+ 5475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.311783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.311852 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.311856 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.311890 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.311931 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":442,"threads_started":1}
I20260928 12:59:58.312526 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.11.11.65:43089)
0928 12:59:58.299371 (+ 13826us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.299378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.299392 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:59:58.299439 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.299442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.299463 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.300626 (+ 1163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:58.300701 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.302058 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.306417 (+ 4359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.307361 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.312251 (+ 4890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.312356 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.312387 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.312446 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13776,"thread_start_us":56,"threads_started":1}
I20260928 12:59:58.312930 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.11.11.67:33139)
0928 12:59:58.290384 (+ 4688us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.290390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.299860 (+ 9470us) server_negotiation.cc:407] Connection header received
0928 12:59:58.299959 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.299962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.299981 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.300096 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:58.302168 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.303560 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.307516 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.308355 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.308433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.308515 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.312680 (+ 4165us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.312735 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
I20260928 12:59:58.313778 22931 tablet_service.cc:1467] Processing CreateTablet for tablet e7aaaea5a936462aaff2d542433b0c06 (DEFAULT_TABLE table=test-table [id=f3cfd240634b43eb95a3f9c93729f7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:58.314205 22931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7aaaea5a936462aaff2d542433b0c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:58.315160 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:43005 (local address 127.0.0.1:57670)
0928 12:59:58.309075 (+ 23566us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.309093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.309142 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.309428 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.309429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.309430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:58.309463 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.309464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.310912 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.310913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.313820 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.313823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.314933 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.314934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.314996 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.314999 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.315046 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.315088 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23549,"thread_start_us":169,"threads_started":1}
I20260928 12:59:58.315450 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.285648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.11.11.66:43005)
0928 12:59:58.292148 (+ 6500us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.292154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.309210 (+ 17056us) server_negotiation.cc:407] Connection header received
0928 12:59:58.309254 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.309257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.309275 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.309344 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:58.309527 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.310808 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.313951 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.314769 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.314844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.314900 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.315312 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.315366 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260928 12:59:58.316079 22812 tablet_service.cc:1467] Processing CreateTablet for tablet e7aaaea5a936462aaff2d542433b0c06 (DEFAULT_TABLE table=test-table [id=f3cfd240634b43eb95a3f9c93729f7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:58.316434 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7aaaea5a936462aaff2d542433b0c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:58.318887 22674 tablet_service.cc:1467] Processing CreateTablet for tablet e7aaaea5a936462aaff2d542433b0c06 (DEFAULT_TABLE table=test-table [id=f3cfd240634b43eb95a3f9c93729f7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:59:58.319247 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7aaaea5a936462aaff2d542433b0c06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:59:58.322933 23034 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: Bootstrap starting.
I20260928 12:59:58.325248 23034 tablet_bootstrap.cc:654] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:58.325663 23034 log.cc:824] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:58.325871 23033 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Bootstrap starting.
I20260928 12:59:58.326545 23034 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: No bootstrap required, opened a new log
I20260928 12:59:58.326617 23034 ts_tablet_manager.cc:1397] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 12:59:58.326789 23032 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Bootstrap starting.
I20260928 12:59:58.327095 23033 tablet_bootstrap.cc:654] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:58.327409 23033 log.cc:824] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:58.327876 23032 tablet_bootstrap.cc:654] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Neither blocks nor log segments found. Creating new log.
I20260928 12:59:58.328193 23032 log.cc:824] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:59:58.329432 23034 raft_consensus.cc:348] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.329607 23034 raft_consensus.cc:374] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:58.329643 23034 raft_consensus.cc:729] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a84fd1ba7a48a28af83c29118d9aef, State: Initialized, Role: FOLLOWER
I20260928 12:59:58.330229 23034 consensus_queue.cc:260] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.331198 23034 ts_tablet_manager.cc:1428] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 12:59:58.331910 23033 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: No bootstrap required, opened a new log
I20260928 12:59:58.331904 23032 tablet_bootstrap.cc:492] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: No bootstrap required, opened a new log
I20260928 12:59:58.331983 23032 ts_tablet_manager.cc:1397] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 12:59:58.332115 23033 ts_tablet_manager.cc:1397] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 12:59:58.332325 22740 heartbeater.cc:503] Master 127.11.11.126:38049 was elected leader, sending a full tablet report...
I20260928 12:59:58.334110 23032 raft_consensus.cc:348] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.334290 23032 raft_consensus.cc:374] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:58.334334 23032 raft_consensus.cc:729] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645fa5f358a64c2a9fe2ced589bf346c, State: Initialized, Role: FOLLOWER
I20260928 12:59:58.334468 23032 consensus_queue.cc:260] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.334739 23032 ts_tablet_manager.cc:1428] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 12:59:58.334893 23014 heartbeater.cc:503] Master 127.11.11.126:38049 was elected leader, sending a full tablet report...
I20260928 12:59:58.335314 23033 raft_consensus.cc:348] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.335505 23033 raft_consensus.cc:374] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:59:58.335561 23033 raft_consensus.cc:729] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16874dc1a160468494e3fdd64b65e1ff, State: Initialized, Role: FOLLOWER
I20260928 12:59:58.335704 23033 consensus_queue.cc:260] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.336079 23033 ts_tablet_manager.cc:1428] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:59:58.336234 22879 heartbeater.cc:503] Master 127.11.11.126:38049 was elected leader, sending a full tablet report...
W20260928 12:59:58.343515 22880 tablet.cc:2369] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:59:58.409621 23040 raft_consensus.cc:482] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:59:58.409783 23040 raft_consensus.cc:504] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.410252 23040 leader_election.cc:288] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a84fd1ba7a48a28af83c29118d9aef (127.11.11.65:43089), 645fa5f358a64c2a9fe2ced589bf346c (127.11.11.67:33139)
I20260928 12:59:58.414903 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.410967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:58013 (local address 127.11.11.65:43089)
0928 12:59:58.411007 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.411009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.411125 (+ 116us) server_negotiation.cc:407] Connection header received
0928 12:59:58.411233 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.411235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.411247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.411280 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:58.411528 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.412317 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.413910 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.414507 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.414556 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.414565 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.414785 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.414833 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:59:58.414938 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.410841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:45437 (local address 127.11.11.67:33139)
0928 12:59:58.410896 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.410899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.411332 (+ 433us) server_negotiation.cc:407] Connection header received
0928 12:59:58.411432 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.411433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.411443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.411472 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:59:58.411654 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.412424 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.414051 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.414577 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.414619 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.414627 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.414834 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.414872 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:59:58.415197 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.410546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43089 (local address 127.11.11.66:58013)
0928 12:59:58.411058 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.411070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.411153 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.411356 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.411357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.411359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:58.411430 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.411433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.412433 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.412434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.413811 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.413814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.414629 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.414629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.414700 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.414704 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.415033 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.415093 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260928 12:59:58.415597 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.410762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33139 (local address 127.11.11.66:45437)
0928 12:59:58.411272 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.411282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:59:58.411356 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:59:58.411535 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:59:58.411536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:59:58.411537 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:59:58.411584 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.411585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.412557 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.412558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.413951 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:59:58.413953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.414692 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.414693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:59:58.414758 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.414761 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:59:58.415466 (+ 705us) client_negotiation.cc:240] Negotiation successful
0928 12:59:58.415509 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260928 12:59:58.415529 22681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7aaaea5a936462aaff2d542433b0c06" candidate_uuid: "16874dc1a160468494e3fdd64b65e1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" is_pre_election: true
I20260928 12:59:58.415721 22681 raft_consensus.cc:2454] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16874dc1a160468494e3fdd64b65e1ff in term 0.
I20260928 12:59:58.415956 22767 leader_election.cc:302] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16874dc1a160468494e3fdd64b65e1ff, a3a84fd1ba7a48a28af83c29118d9aef; no voters:
I20260928 12:59:58.416117 23040 raft_consensus.cc:2781] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:59:58.416169 23040 raft_consensus.cc:482] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:59:58.416124 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7aaaea5a936462aaff2d542433b0c06" candidate_uuid: "16874dc1a160468494e3fdd64b65e1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645fa5f358a64c2a9fe2ced589bf346c" is_pre_election: true
I20260928 12:59:58.416193 23040 raft_consensus.cc:3037] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:58.416272 22963 raft_consensus.cc:2454] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16874dc1a160468494e3fdd64b65e1ff in term 0.
I20260928 12:59:58.417039 23040 raft_consensus.cc:504] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.417194 23040 leader_election.cc:288] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [CANDIDATE]: Term 1 election: Requested vote from peers a3a84fd1ba7a48a28af83c29118d9aef (127.11.11.65:43089), 645fa5f358a64c2a9fe2ced589bf346c (127.11.11.67:33139)
I20260928 12:59:58.417364 22681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7aaaea5a936462aaff2d542433b0c06" candidate_uuid: "16874dc1a160468494e3fdd64b65e1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a84fd1ba7a48a28af83c29118d9aef"
I20260928 12:59:58.417377 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7aaaea5a936462aaff2d542433b0c06" candidate_uuid: "16874dc1a160468494e3fdd64b65e1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645fa5f358a64c2a9fe2ced589bf346c"
I20260928 12:59:58.417438 22681 raft_consensus.cc:3037] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:58.418398 22681 raft_consensus.cc:2454] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16874dc1a160468494e3fdd64b65e1ff in term 1.
I20260928 12:59:58.418582 22767 leader_election.cc:302] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16874dc1a160468494e3fdd64b65e1ff, a3a84fd1ba7a48a28af83c29118d9aef; no voters:
I20260928 12:59:58.418658 23040 raft_consensus.cc:2781] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:59:58.417438 22963 raft_consensus.cc:3037] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:59:58.418809 23040 raft_consensus.cc:686] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 1 LEADER]: Becoming Leader. State: Replica: 16874dc1a160468494e3fdd64b65e1ff, State: Running, Role: LEADER
I20260928 12:59:58.418898 23040 consensus_queue.cc:237] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } }
I20260928 12:59:58.419759 22963 raft_consensus.cc:2454] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16874dc1a160468494e3fdd64b65e1ff in term 1.
I20260928 12:59:58.419764 22530 catalog_manager.cc:5495] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 16874dc1a160468494e3fdd64b65e1ff (127.11.11.66). New cstate: current_term: 1 leader_uuid: "16874dc1a160468494e3fdd64b65e1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16874dc1a160468494e3fdd64b65e1ff" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 43005 } health_report { overall_health: HEALTHY } } }
I20260928 12:59:58.439728 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.436114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.11.11.66:43005)
0928 12:59:58.436161 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:59:58.436163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:59:58.436229 (+ 66us) server_negotiation.cc:407] Connection header received
0928 12:59:58.436319 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:59:58.436320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:59:58.436345 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:59:58.436440 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:59:58.436831 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.437370 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.438497 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:59:58.438799 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:59:58.438863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:59:58.439112 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:59:58.439265 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:59:58.439368 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:59:58.439622 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 12:59:58.439645 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:59:58.442857 22681 raft_consensus.cc:1261] T e7aaaea5a936462aaff2d542433b0c06 P a3a84fd1ba7a48a28af83c29118d9aef [term 1 FOLLOWER]: Refusing update from remote peer 16874dc1a160468494e3fdd64b65e1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:59:58.442886 22963 raft_consensus.cc:1261] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 1 FOLLOWER]: Refusing update from remote peer 16874dc1a160468494e3fdd64b65e1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:59:58.443228 23040 consensus_queue.cc:1035] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a84fd1ba7a48a28af83c29118d9aef" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 43089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:59:58.443423 23040 consensus_queue.cc:1035] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "645fa5f358a64c2a9fe2ced589bf346c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:59:58.447126 23046 mvcc.cc:205] Tried to move back new op lower bound from 7334299232018124800 to 7334299231924219904. Current Snapshot: MvccSnapshot[applied={T|T < 7334299232018124800}]
I20260928 12:59:58.452224 23047 mvcc.cc:205] Tried to move back new op lower bound from 7334299232018124800 to 7334299231924219904. Current Snapshot: MvccSnapshot[applied={T|T < 7334299232018124800}]
I20260928 12:59:58.452203 22529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:59:58.452320 22529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:59:58.452620 23048 mvcc.cc:205] Tried to move back new op lower bound from 7334299232018124800 to 7334299231924219904. Current Snapshot: MvccSnapshot[applied={T|T < 7334299232018124800}]
I20260928 12:59:58.460392 22529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b17f330abc745ba957dc56e61e32750: Sending DeleteTablet for 3 replicas of tablet e7aaaea5a936462aaff2d542433b0c06
I20260928 12:59:58.461329 22674 tablet_service.cc:1514] Processing DeleteTablet for tablet e7aaaea5a936462aaff2d542433b0c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:58 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59436
I20260928 12:59:58.461733 22931 tablet_service.cc:1514] Processing DeleteTablet for tablet e7aaaea5a936462aaff2d542433b0c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:58 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260928 12:59:58.462036 22812 tablet_service.cc:1514] Processing DeleteTablet for tablet e7aaaea5a936462aaff2d542433b0c06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:59:58 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:57670
I20260928 12:59:58.462652 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 22611
I20260928 12:59:58.467519 23058 tablet_replica.cc:330] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: stopping tablet replica
I20260928 12:59:58.467690 23058 raft_consensus.cc:2229] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:59:58.467836 23058 raft_consensus.cc:2258] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:59:58.471652 23058 ts_tablet_manager.cc:1905] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:59:58.476250 23057 tablet_replica.cc:330] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: stopping tablet replica
I20260928 12:59:58.476421 23057 raft_consensus.cc:2229] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:59:58.476634 23057 raft_consensus.cc:2258] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:59:58.477088 23058 ts_tablet_manager.cc:1918] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:59:58.477191 23058 log.cc:1196] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/e7aaaea5a936462aaff2d542433b0c06
I20260928 12:59:58.477531 23058 ts_tablet_manager.cc:1939] T e7aaaea5a936462aaff2d542433b0c06 P 16874dc1a160468494e3fdd64b65e1ff: Deleting consensus metadata
I20260928 12:59:58.478093 22514 catalog_manager.cc:4841] TS 16874dc1a160468494e3fdd64b65e1ff (127.11.11.66:43005): tablet e7aaaea5a936462aaff2d542433b0c06 (table test-table [id=f3cfd240634b43eb95a3f9c93729f7fd]) successfully deleted
I20260928 12:59:58.480159 23057 ts_tablet_manager.cc:1905] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:59:58.481487 22515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:43089 (error 108)
I20260928 12:59:58.482340 23057 ts_tablet_manager.cc:1918] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:59:58.482434 23057 log.cc:1196] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/e7aaaea5a936462aaff2d542433b0c06
I20260928 12:59:58.483268 23057 ts_tablet_manager.cc:1939] T e7aaaea5a936462aaff2d542433b0c06 P 645fa5f358a64c2a9fe2ced589bf346c: Deleting consensus metadata
I20260928 12:59:58.483819 22514 catalog_manager.cc:4841] TS 645fa5f358a64c2a9fe2ced589bf346c (127.11.11.67:33139): tablet e7aaaea5a936462aaff2d542433b0c06 (table test-table [id=f3cfd240634b43eb95a3f9c93729f7fd]) successfully deleted
I20260928 12:59:58.486557 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 22748
I20260928 12:59:58.492815 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:59:58.492661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:43089 (local address 127.0.0.1:59444)
0928 12:59:58.492719 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:59:58.492767 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 12:59:58.492981 22515 catalog_manager.cc:4584] TS a3a84fd1ba7a48a28af83c29118d9aef (127.11.11.65:43089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7aaaea5a936462aaff2d542433b0c06: Network error: Client connection negotiation failed: client connection to 127.11.11.65:43089: connect: Connection refused (error 111)
I20260928 12:59:58.493597 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 22885
I20260928 12:59:58.499742 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 22498
Sep 28 12:59:58 dist-test-slave-8sjz krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600397, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 12:59:58.505693 11309 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:59:58.850492 11309 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:59:58.850632 11309 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:59:59.195657 11309 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:59:59.195 UTC [22084] LOG: received smart shutdown request
2026-09-28 12:59:59.199 UTC [22084] LOG: background worker "logical replication launcher" (PID 22093) exited with exit code 1
2026-09-28 12:59:59.199 UTC [22088] LOG: shutting down
2026-09-28 12:59:59.369 UTC [22084] LOG: database system is shut down
2026-09-28T12:59:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48424 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:59:59 dist-test-slave-8sjz krb5kdc[23068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:59:59 dist-test-slave-8sjz krb5kdc[23068](info): set up 2 sockets
Sep 28 12:59:59 dist-test-slave-8sjz krb5kdc[23068](info): commencing operation
krb5kdc: starting...
W20260928 13:00:00.517318 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:00:00 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:00:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangeradmin_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangeradmin_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangerlookup_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangerlookup_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
I20260928 13:00:00.691406 11309 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNYzOw4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/postgres -l logfile start
2026-09-28 13:00:03.368 UTC [23098] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 13:00:03.368 UTC [23098] LOG: listening on IPv4 address "127.11.11.84", port 46979
2026-09-28 13:00:03.373 UTC [23098] LOG: listening on Unix socket "/tmp/.s.PGSQL.46979"
2026-09-28 13:00:03.409 UTC [23101] LOG: database system was shut down at 2026-09-28 13:00:01 UTC
2026-09-28 13:00:03.414 UTC [23098] LOG: database system is ready to accept connections
I20260928 13:00:04.377080 11309 mini_postgres.cc:96] Postgres bound to 46979
2026-09-28 13:00:04.383 UTC [23109] FATAL: database "slave" does not exist
127.11.11.84:46979 - accepting connections
I20260928 13:00:04.383893 11309 mini_ranger.cc:161] Starting Ranger...
I20260928 13:00:04.399165 11309 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 13:00:04.935813 11309 mini_ranger.cc:87] Created ranger Postgres database
I20260928 13:00:04.935906 11309 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin
2026-09-28 13:00:05,218 [I] DB FLAVOR :POSTGRES
2026-09-28 13:00:05,218 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:00:05,219 [I] Checking connection..
2026-09-28 13:00:05,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 13:00:05,458 [I] Checking connection passed.
2026-09-28 13:00:05,458 [I] --------- Verifying version history table ---------
2026-09-28 13:00:05,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:00:05,710 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:00:05,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:00:05,959 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:00:05,960 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 13:00:05,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 13:00:06.225 UTC [23173] WARNING: there is no transaction in progress
2026-09-28 13:00:06,232 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 13:00:06,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:00:06,474 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 13:00:06,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:00:06,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 13:00:06,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 13:00:06,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8sjz.c.gcp-upstream.internal','N') ;"
2026-09-28 13:00:07,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:00:07,483 [I] Table x_portal_user does not exist in database ranger
2026-09-28 13:00:07,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:00:07,723 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 13:00:07,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:00:07,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:00:08,214 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 13:00:08,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 13:00:09.235 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.252 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.270 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.288 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.307 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.326 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.388 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.396 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.412 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.423 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.434 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.449 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.459 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.466 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.473 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:09.481 UTC [23281] WARNING: there is no transaction in progress
2026-09-28 13:00:10,174 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 13:00:10,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 13:00:10,430 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 13:00:10,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:00:10,695 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 13:00:10,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:00:10,966 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 13:00:10,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:00:11,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:00:11,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8sjz.c.gcp-upstream.internal';"
2026-09-28 13:00:11,707 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 13:00:11,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 13:00:11,943 [I] DB_PATCHES have already been applied
I20260928 13:00:11.949688 11309 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNYzOw4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 13:00:11.949756 11309 mini_ranger.cc:193] Using host: 127.11.11.84
I20260928 13:00:11.952869 11309 mini_ranger.cc:227] Ranger admin URL: http://127.11.11.84:60411
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 1:00:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 13:00:13.071543 11309 mini_ranger.cc:160] Time spent starting Ranger: real 8.688s user 0.000s sys 0.009s
Sep 28, 2026 1:00:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 1:00:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 1:00:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 1:00:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 1:00:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 13:00:35.721186 11309 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.11.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.11.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangerkms_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.11.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/rangerkms_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.11.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.11.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.11.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/HTTP_127.11.11.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 13:00:35 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 13:00:35.810583 11309 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 13:00:35.820794 11309 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 13:00:36.756759 11309 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 13:00:36.756860 11309 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms
I20260928 13:00:36.756877 11309 mini_ranger_kms.cc:227] Using postgres at 127.11.11.84:46979
2026-09-28 13:00:36,922 [I] DB FLAVOR :POSTGRES
2026-09-28 13:00:36,923 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:00:36,923 [I] Checking connection
2026-09-28 13:00:36,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 13:00:37,184 [I] connection success
2026-09-28 13:00:37,184 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 13:00:37,184 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 13:00:37,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 13:00:37,431 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 13:00:37,431 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:00:37,431 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 13:00:37,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.11.84:46979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 13:00:37,693 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 13:00:38.189170 11309 mini_ranger_kms.cc:310] Created kms service
I20260928 13:00:38.377656 11309 mini_ranger_kms.cc:326] Created kudu user
I20260928 13:00:38.421008 11309 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 13:00:38 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600435, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:38.837390 11309 mini_ranger_kms.cc:384] Added ranger policy
I20260928 13:00:38.837599 11309 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNYzOw4/build/release/bin/postgresql.jar:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNYzOw4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 13:00:38.837641 11309 mini_ranger_kms.cc:240] Using host: 127.11.11.84
I20260928 13:00:38.840909 11309 mini_ranger_kms.cc:276] Ranger KMS PID: 23462
I20260928 13:00:38.840993 11309 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.11.84:48977
Sep 28, 2026 1:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 1:00:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 1:00:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 1:00:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 13:00:41.101824 11309 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.291s user 0.000s sys 0.148s
I20260928 13:00:41.101935 11309 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 13:00:41.101981 11309 mini_ranger_kms.cc:401] 127.11.11.84:48977/kms/v1/keys
Sep 28 13:00:41 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790600441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 13:00:41 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
Sep 28 13:00:41 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600441, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.11.84@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
Sep 28 13:00:44 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:00:44.234973 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34473
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:34473
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:44.346765 23512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:44.346992 23512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:44.347015 23512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:44.348323 23512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:44.348377 23512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:44.348392 23512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:44.348404 23512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:44.348416 23512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:44.350069 23512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.11.84:48977/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:34473
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:44.350356 23512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:44.350602 23512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:44.356282 23518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:44.356201 23520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:44.356210 23517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:44.356464 23512 server_base.cc:1034] running on GCE node
I20260928 13:00:44.356917 23512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:44.357281 23512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:44.358456 23512 hybrid_clock.cc:648] HybridClock initialized: now 1790600444358437 us; error 36 us; skew 500 ppm
Sep 28 13:00:44 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600444, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:44.366390 23512 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:44.367000 23512 webserver.cc:460] Webserver started at http://127.11.11.126:46527/ using document root <none> and password file <none>
I20260928 13:00:44.367254 23512 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:44.367326 23512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:44.367489 23512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:00:44 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600444, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:44.429852 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "d843e0b07f5a4e589a08a2581061a759"
format_stamp: "Formatted at 2026-09-28 13:00:44 on dist-test-slave-8sjz"
server_key: "5a34ef3424127f979ed7feca3da3f8d5"
server_key_iv: "7634f14202ce3fd7b0f30f830e8fc80d"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:44.430377 23512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "d843e0b07f5a4e589a08a2581061a759"
format_stamp: "Formatted at 2026-09-28 13:00:44 on dist-test-slave-8sjz"
server_key: "5a34ef3424127f979ed7feca3da3f8d5"
server_key_iv: "7634f14202ce3fd7b0f30f830e8fc80d"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:44.431891 23512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:44.446452 23527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:44.446779 23512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:44.446900 23512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "d843e0b07f5a4e589a08a2581061a759"
format_stamp: "Formatted at 2026-09-28 13:00:44 on dist-test-slave-8sjz"
server_key: "5a34ef3424127f979ed7feca3da3f8d5"
server_key_iv: "7634f14202ce3fd7b0f30f830e8fc80d"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:44.447005 23512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:44.470186 23512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:44.472573 23512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:44.472743 23512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:44.478219 23512 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:34473
I20260928 13:00:44.478246 23579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:34473 every 8 connection(s)
I20260928 13:00:44.478878 23512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:44.479768 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:44.482504 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 23512
I20260928 13:00:44.482630 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:00:44.482806 11309 external_mini_cluster.cc:1428] Setting key 701ec51e0e3855bdb4fdd4e01789d2ff
I20260928 13:00:44.484783 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Bootstrap starting.
I20260928 13:00:44.485546 23580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:44.485920 23580 log.cc:824] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:44.486797 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: No bootstrap required, opened a new log
I20260928 13:00:44.489143 23580 raft_consensus.cc:348] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:44.489356 23580 raft_consensus.cc:374] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:44.489427 23580 raft_consensus.cc:729] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843e0b07f5a4e589a08a2581061a759, State: Initialized, Role: FOLLOWER
I20260928 13:00:44.489579 23580 consensus_queue.cc:260] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:44.489693 23580 raft_consensus.cc:388] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:44.489755 23580 raft_consensus.cc:482] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:44.489825 23580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:44.490410 23580 raft_consensus.cc:504] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:44.490542 23580 leader_election.cc:302] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d843e0b07f5a4e589a08a2581061a759; no voters:
I20260928 13:00:44.490756 23580 leader_election.cc:288] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:00:44.490792 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:44.491087 23580 sys_catalog.cc:564] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 13:00:44 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:44.491582 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 LEADER]: Becoming Leader. State: Replica: d843e0b07f5a4e589a08a2581061a759, State: Running, Role: LEADER
I20260928 13:00:44.491673 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:44.492018 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d843e0b07f5a4e589a08a2581061a759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } } }
I20260928 13:00:44.492026 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d843e0b07f5a4e589a08a2581061a759. Latest consensus state: current_term: 1 leader_uuid: "d843e0b07f5a4e589a08a2581061a759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } } }
I20260928 13:00:44.492170 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:44.492417 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:44.492785 23591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:44.493355 23591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:44.494980 23591 catalog_manager.cc:1347] Generated new cluster ID: 6d383837d79447c29149ccf75f56dfa2
I20260928 13:00:44.495424 23580 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 13:00:44.495078 23591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:44.497486 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:44.483609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.11.11.126:34473)
0928 13:00:44.483815 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:44.483825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:44.483859 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:00:44.484442 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:44.484452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:44.484525 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:44.484692 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:44.485629 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:44.486981 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:44.488273 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:44.488912 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:44.489031 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:44.493357 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:44.493363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:44.493363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:44.493377 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:44.495651 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:44.496116 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:44.496120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:44.496120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:44.496255 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:44.496493 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:44.496494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:44.496495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:44.496853 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:44.497015 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:44.497100 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 13:00:44.497239 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260928 13:00:44.536862 23591 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:00:44.537760 23591 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:44.554057 23591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Generated new TSK 0
I20260928 13:00:44.554385 23591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:00:45.299758 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:45.411281 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:45.411584 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:45.411623 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:45.413415 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:45.413481 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:45.413524 23625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:00:45.415701 23625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:45.415969 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:45.416301 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:45.422590 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.422658 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.422590 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:45.423228 23625 server_base.cc:1034] running on GCE node
I20260928 13:00:45.423440 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:45.423727 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:45.424908 23625 hybrid_clock.cc:648] HybridClock initialized: now 1790600445424886 us; error 34 us; skew 500 ppm
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:45.428026 23625 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:00:45.428524 23625 webserver.cc:460] Webserver started at http://127.11.11.65:45541/ using document root <none> and password file <none>
I20260928 13:00:45.428782 23625 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:45.428853 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:45.429004 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:45.443884 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "aa97b7e0e3d347cd86927727690a37ff"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "f9452b75c86da099c4183a60d11b5d00"
server_key_iv: "6cd24b3432eb45542022fe97a1f27cbe"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.444470 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "aa97b7e0e3d347cd86927727690a37ff"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "f9452b75c86da099c4183a60d11b5d00"
server_key_iv: "6cd24b3432eb45542022fe97a1f27cbe"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.448414 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 13:00:45.466892 23642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.467083 23625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:45.467171 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "aa97b7e0e3d347cd86927727690a37ff"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "f9452b75c86da099c4183a60d11b5d00"
server_key_iv: "6cd24b3432eb45542022fe97a1f27cbe"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.467283 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:45.490474 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:45.492843 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:45.493014 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:45.493302 23625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:45.493700 23625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:45.493752 23625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.493810 23625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:45.493852 23625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.501427 23625 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38357
I20260928 13:00:45.501515 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38357 every 8 connection(s)
I20260928 13:00:45.501973 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:00:45.507191 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 23625
I20260928 13:00:45.507290 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:00:45.507457 11309 external_mini_cluster.cc:1428] Setting key d36f015fe2478ab3ee32104afb31772a
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:45.515951 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:45.503096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59713 (local address 127.11.11.126:34473)
0928 13:00:45.503898 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:45.503901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:45.503909 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:00:45.503936 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:45.503937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:45.503950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:45.504816 (+ 866us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:45.504863 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.505670 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.507234 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.507538 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.508093 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:45.511906 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:45.511908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:45.511908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:45.511916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:45.513943 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:45.514574 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:45.514576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:45.514576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:45.514648 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:45.515028 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:45.515030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:45.515030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:45.515202 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:45.515280 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:45.515730 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 13:00:45.515843 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":117,"threads_started":1}
I20260928 13:00:45.516186 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:45.502784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.65:59713)
0928 13:00:45.502981 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:45.503031 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:45.503686 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:45.504155 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:45.504160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:45.504423 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:45.504714 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:45.504721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.505790 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.505792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:45.507095 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:45.507099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.507655 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.507656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:45.507805 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:45.510021 (+ 2216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:45.510032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:45.511777 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:45.514098 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:45.514108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:45.514111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:45.514484 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:45.514731 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:45.514732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:45.514733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:45.514941 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:45.515298 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:45.515318 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:45.515555 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 13:00:45.515690 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 13:00:45.515885 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 13:00:45.517021 23756 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:45.517128 23756 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 13:00:45.517369 23756 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:45.517966 23544 ts_manager.cc:194] Registered new tserver with Master: aa97b7e0e3d347cd86927727690a37ff (127.11.11.65:38357)
I20260928 13:00:45.518913 23544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:59713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:00:45.535389 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:45.639844 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:45.640115 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:45.640158 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:45.641427 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:45.641503 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:45.641552 23762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:00:45.643115 23762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:45.643414 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:45.643679 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:45.649430 23772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.649814 23769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.649791 23770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:45.650787 23762 server_base.cc:1034] running on GCE node
I20260928 13:00:45.650998 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:45.651232 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:45.652384 23762 hybrid_clock.cc:648] HybridClock initialized: now 1790600445652367 us; error 32 us; skew 500 ppm
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:45.655385 23762 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:00:45.655979 23762 webserver.cc:460] Webserver started at http://127.11.11.66:36761/ using document root <none> and password file <none>
I20260928 13:00:45.656271 23762 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:45.656345 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:45.656493 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:45.669049 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "dcb12b03bd914fa487b2a5c5704f0fe4"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "e9bb00a2aac1eb5bb39519ae0f355171"
server_key_iv: "d9283bb1a84ba345cbb619d9117e11c8"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.669467 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "dcb12b03bd914fa487b2a5c5704f0fe4"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "e9bb00a2aac1eb5bb39519ae0f355171"
server_key_iv: "d9283bb1a84ba345cbb619d9117e11c8"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.671646 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 13:00:45.695266 23779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.695554 23762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 13:00:45.695662 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "dcb12b03bd914fa487b2a5c5704f0fe4"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "e9bb00a2aac1eb5bb39519ae0f355171"
server_key_iv: "d9283bb1a84ba345cbb619d9117e11c8"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.695760 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:45.718978 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:45.721503 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:45.721669 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:45.721973 23762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:45.722419 23762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:45.722478 23762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.722513 23762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:45.722575 23762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.730311 23762 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:34719
I20260928 13:00:45.730453 23892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:34719 every 8 connection(s)
I20260928 13:00:45.730911 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:00:45.732862 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 23762
I20260928 13:00:45.732970 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:00:45.733165 11309 external_mini_cluster.cc:1428] Setting key c3912a8880ebc17199bf3384251f7b5b
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:45.743405 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:45.731778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:55123 (local address 127.11.11.126:34473)
0928 13:00:45.731842 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:45.731845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:45.732121 (+ 276us) server_negotiation.cc:407] Connection header received
0928 13:00:45.732877 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:45.732879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:45.732896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:45.732931 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:45.734526 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.735314 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.736785 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.737084 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.737130 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:45.739894 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:45.739896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:45.739896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:45.739905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:45.741727 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:45.742246 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:45.742248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:45.742248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:45.742310 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:45.742604 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:45.742605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:45.742605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:45.742743 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:45.742786 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:45.743208 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 13:00:45.743307 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 13:00:45.744304 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:45.731782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.66:55123)
0928 13:00:45.731963 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:45.732009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:45.732716 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:45.733154 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:45.733158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:45.734093 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:45.734406 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:45.734414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.735447 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.735448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:45.736682 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:45.736686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:45.737242 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:45.737243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:45.737385 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:45.738029 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:45.738040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:45.739763 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:45.741845 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:45.741852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:45.741855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:45.742158 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:45.742382 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:45.742382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:45.742383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:45.742516 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:45.742848 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:45.742854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:45.743029 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 13:00:45.743833 (+ 804us) client_negotiation.cc:240] Negotiation successful
0928 13:00:45.744018 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260928 13:00:45.745759 23893 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:45.745865 23893 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:45.746094 23893 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:45.746505 23544 ts_manager.cc:194] Registered new tserver with Master: dcb12b03bd914fa487b2a5c5704f0fe4 (127.11.11.66:34719)
I20260928 13:00:45.747016 23544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:55123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:00:45.775050 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:45.901609 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:45.901999 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:45.902105 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:45.904284 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:45.904418 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:45.904518 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:00:45.907023 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:45.907582 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:45.907939 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:45.915202 23905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.915681 23904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:45.916584 23907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:45.916864 23899 server_base.cc:1034] running on GCE node
I20260928 13:00:45.917181 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:45.917498 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:45.918677 23899 hybrid_clock.cc:648] HybridClock initialized: now 1790600445918654 us; error 35 us; skew 500 ppm
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:45.922264 23899 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:00:45.922873 23899 webserver.cc:460] Webserver started at http://127.11.11.67:35835/ using document root <none> and password file <none>
I20260928 13:00:45.923174 23899 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:45.923243 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:45.923393 23899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:00:45 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:45.951788 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "20b85d6cf2d248ad9fd45cc7cda5d363"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "7e986beb6321ddc532f35ecc66fb93e9"
server_key_iv: "ce4d6c0796adbfa63f6dfa35f44481d2"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.952345 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "20b85d6cf2d248ad9fd45cc7cda5d363"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "7e986beb6321ddc532f35ecc66fb93e9"
server_key_iv: "ce4d6c0796adbfa63f6dfa35f44481d2"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.955096 23899 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 13:00:45.972204 23914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.972489 23899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 13:00:45.972584 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "20b85d6cf2d248ad9fd45cc7cda5d363"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "7e986beb6321ddc532f35ecc66fb93e9"
server_key_iv: "ce4d6c0796adbfa63f6dfa35f44481d2"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:45.972678 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:45.991135 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:45.993671 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:45.993898 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:45.994166 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:45.994571 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:45.994660 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:45.994729 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:45.994766 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.001974 23899 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:43167
I20260928 13:00:46.002112 24027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:43167 every 8 connection(s)
I20260928 13:00:46.002509 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:00:46.004959 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 23899
I20260928 13:00:46.005059 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:00:46.005241 11309 external_mini_cluster.cc:1428] Setting key 54b241c1490bf7ef18d974e64cd1b9c3
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600445, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:46.013981 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.003343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:38729 (local address 127.11.11.126:34473)
0928 13:00:46.003390 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:46.003393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:46.003680 (+ 287us) server_negotiation.cc:407] Connection header received
0928 13:00:46.004361 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:46.004363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:46.004385 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:46.004460 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:46.005184 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.005920 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.007427 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.007734 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.007816 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.010487 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:46.010488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:46.010489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:46.010498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:46.012253 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.012797 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.012798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.012798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.012864 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.013182 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.013183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.013183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.013328 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:46.013397 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:46.013783 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 13:00:46.013885 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:00:46.014171 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.003323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.67:38729)
0928 13:00:46.003518 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:46.003567 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:46.004248 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:46.004541 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:46.004545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:46.004805 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:46.005074 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.005080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.006034 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.006035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.007333 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.007336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.007856 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.007857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.007995 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.008649 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:46.008658 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:46.010364 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:46.012343 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.012351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.012354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.012698 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.012941 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.012942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.012942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.013099 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.013409 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:46.013416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:46.013615 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 13:00:46.013741 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 13:00:46.013917 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260928 13:00:46.014830 24028 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:46.014947 24028 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.015197 24028 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:46.015708 23544 ts_manager.cc:194] Registered new tserver with Master: 20b85d6cf2d248ad9fd45cc7cda5d363 (127.11.11.67:43167)
I20260928 13:00:46.016268 23544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:38729
I20260928 13:00:46.016901 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:00:46.017607 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 23625
I20260928 13:00:46.024333 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 23762
I20260928 13:00:46.035830 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 23899
I20260928 13:00:46.043217 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 23512
I20260928 13:00:46.049808 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34473
--webserver_interface=127.11.11.126
--webserver_port=46527
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:34473
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:46.134347 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:46.134572 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:46.134596 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:46.135810 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:46.135864 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:46.135878 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:46.135890 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:46.135903 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:46.137638 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:34473
--ranger_config_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:46.137890 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:46.138144 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:46.143250 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.143126 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.143126 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:46.143486 24031 server_base.cc:1034] running on GCE node
I20260928 13:00:46.143828 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:46.144194 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:46.145363 24031 hybrid_clock.cc:648] HybridClock initialized: now 1790600446145350 us; error 30 us; skew 500 ppm
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:46.148001 24031 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:46.148597 24031 webserver.cc:460] Webserver started at http://127.11.11.126:46527/ using document root <none> and password file <none>
I20260928 13:00:46.148852 24031 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:46.148927 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:46.207147 24031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:46.207926 24047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.208113 24031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:00:46.208220 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "d843e0b07f5a4e589a08a2581061a759"
format_stamp: "Formatted at 2026-09-28 13:00:44 on dist-test-slave-8sjz"
server_key: "5a34ef3424127f979ed7feca3da3f8d5"
server_key_iv: "7634f14202ce3fd7b0f30f830e8fc80d"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:46.208599 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:46.223634 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:46.225615 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:46.225771 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:46.230229 24031 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:34473
I20260928 13:00:46.230268 24099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:34473 every 8 connection(s)
I20260928 13:00:46.230661 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:46.231859 24100 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:00:46.232594 24100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Bootstrap starting.
I20260928 13:00:46.237022 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24031
I20260928 13:00:46.237421 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:38357
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:34473
--webserver_port=45541
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:46.241816 24100 log.cc:824] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:46.243568 24100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:00:46.243868 24100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Bootstrap complete.
I20260928 13:00:46.246338 24100 raft_consensus.cc:348] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:46.246577 24100 raft_consensus.cc:729] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843e0b07f5a4e589a08a2581061a759, State: Initialized, Role: FOLLOWER
I20260928 13:00:46.246747 24100 consensus_queue.cc:260] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:46.246822 24100 raft_consensus.cc:388] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:46.246891 24100 raft_consensus.cc:482] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:46.246938 24100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:00:46.247978 24100 raft_consensus.cc:504] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:46.248152 24100 leader_election.cc:302] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d843e0b07f5a4e589a08a2581061a759; no voters:
I20260928 13:00:46.248329 24100 leader_election.cc:288] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:00:46.248366 24104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:00:46.248539 24104 raft_consensus.cc:686] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [term 2 LEADER]: Becoming Leader. State: Replica: d843e0b07f5a4e589a08a2581061a759, State: Running, Role: LEADER
I20260928 13:00:46.248611 24100 sys_catalog.cc:564] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:46.248694 24104 consensus_queue.cc:237] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } }
I20260928 13:00:46.248957 24104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d843e0b07f5a4e589a08a2581061a759. Latest consensus state: current_term: 2 leader_uuid: "d843e0b07f5a4e589a08a2581061a759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } } }
I20260928 13:00:46.249039 24104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:46.248960 24105 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d843e0b07f5a4e589a08a2581061a759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843e0b07f5a4e589a08a2581061a759" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34473 } } }
I20260928 13:00:46.249212 24105 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:46.250113 24100 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 13:00:46.252177 24110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:46.253153 24110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:46.253458 24110 catalog_manager.cc:1259] Loaded cluster ID: 6d383837d79447c29149ccf75f56dfa2
I20260928 13:00:46.253546 24110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:46.255672 24110 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:46.255923 24110 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Loaded TSK: 0
I20260928 13:00:46.256341 24110 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:00:46.346391 24102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:46.346707 24102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:46.346769 24102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:46.348351 24102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:46.348420 24102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:46.348469 24102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:00:46.350414 24102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:38357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=45541
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:46.350703 24102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:46.350955 24102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:46.355999 24140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.356004 24138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.356456 24137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:46.357101 24102 server_base.cc:1034] running on GCE node
I20260928 13:00:46.357391 24102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:46.357666 24102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:46.363509 24102 hybrid_clock.cc:648] HybridClock initialized: now 1790600446363489 us; error 32 us; skew 500 ppm
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:46.366930 24102 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:00:46.367570 24102 webserver.cc:460] Webserver started at http://127.11.11.65:45541/ using document root <none> and password file <none>
I20260928 13:00:46.367872 24102 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:46.367970 24102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:46.397397 24102 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 13:00:46.401656 24148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.402177 24102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260928 13:00:46.402326 24102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "aa97b7e0e3d347cd86927727690a37ff"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "f9452b75c86da099c4183a60d11b5d00"
server_key_iv: "6cd24b3432eb45542022fe97a1f27cbe"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:46.402871 24102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:46.425452 24102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:46.427974 24102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:46.428227 24102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:46.428553 24102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:46.429024 24102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:46.429085 24102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.429119 24102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:46.429191 24102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.437510 24102 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38357
I20260928 13:00:46.437619 24261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38357 every 8 connection(s)
I20260928 13:00:46.438051 24102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:00:46.444092 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24102
I20260928 13:00:46.444491 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:34719
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:34473
--webserver_port=36761
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:46.451365 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.439174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.65:59009)
0928 13:00:46.439364 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:46.439424 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:46.440024 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:46.440655 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:46.440659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:46.440873 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:46.441078 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.441083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.442245 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.442246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.443068 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.443071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.443844 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.443845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.443988 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.444913 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:46.444922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:46.446316 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:46.448960 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.448968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.448971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.449353 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.449693 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.449694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.449695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.449890 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.450528 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:46.450536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:46.450730 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 13:00:46.450873 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 13:00:46.451072 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260928 13:00:46.451385 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.439298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59009 (local address 127.11.11.126:34473)
0928 13:00:46.440214 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:46.440225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:46.440270 (+ 45us) server_negotiation.cc:407] Connection header received
0928 13:00:46.440407 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:46.440412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:46.440456 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:46.440606 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:46.441209 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.442125 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.443208 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.443694 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.443828 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.446474 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:46.446480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:46.446480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:46.446497 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:46.448838 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.449464 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.449468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.449469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.449606 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.449971 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.449972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.449972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.450431 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:46.450537 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:46.450958 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 13:00:46.451109 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":95,"threads_started":1}
I20260928 13:00:46.452620 24262 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:46.452733 24262 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.452960 24262 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.454157 24262 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:34473 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:00:46.454236 24262 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.454313 24262 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:46.454973 24262 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.455056 24262 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.455472 24262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:00:46.561484 24266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:46.561888 24266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:46.561985 24266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:46.563948 24266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:46.564124 24266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:46.564215 24266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:00:46.566644 24266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:34719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=36761
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:46.566948 24266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:46.567319 24266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:46.573251 24271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.573421 24272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.573547 24274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:46.573848 24266 server_base.cc:1034] running on GCE node
I20260928 13:00:46.574065 24266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:46.574337 24266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:46.575510 24266 hybrid_clock.cc:648] HybridClock initialized: now 1790600446575495 us; error 30 us; skew 500 ppm
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:46.578353 24266 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:00:46.578933 24266 webserver.cc:460] Webserver started at http://127.11.11.66:36761/ using document root <none> and password file <none>
I20260928 13:00:46.579200 24266 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:46.579298 24266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:46.599370 24266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:46.600704 24282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.600942 24266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:00:46.601082 24266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "dcb12b03bd914fa487b2a5c5704f0fe4"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "e9bb00a2aac1eb5bb39519ae0f355171"
server_key_iv: "d9283bb1a84ba345cbb619d9117e11c8"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:46.601639 24266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:46.618783 24266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:46.621388 24266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:46.621615 24266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:46.621944 24266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:46.622382 24266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:46.622463 24266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.622534 24266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:46.622578 24266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.630443 24266 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:34719
I20260928 13:00:46.630551 24395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:34719 every 8 connection(s)
I20260928 13:00:46.630997 24266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:00:46.633781 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24266
I20260928 13:00:46.634075 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.11.84:48977/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:43167
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:34473
--webserver_port=35835
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:46.642262 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.632151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:53951 (local address 127.11.11.126:34473)
0928 13:00:46.632198 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:46.632200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:46.632214 (+ 14us) server_negotiation.cc:407] Connection header received
0928 13:00:46.632947 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:46.632948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:46.632980 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:46.633049 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:46.633816 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.634584 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.636182 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.636495 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.636548 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.639263 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:46.639265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:46.639265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:46.639275 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:46.640772 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.641232 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.641234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.641234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.641295 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.641572 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.641573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.641574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.641700 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:46.641778 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:46.642069 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 13:00:46.642158 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:00:46.642467 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.631859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.66:53951)
0928 13:00:46.632080 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:46.632133 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:46.632824 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:46.633129 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:46.633134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:46.633381 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:46.633678 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.633685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.634706 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.634707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.636017 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.636021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.636667 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.636668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.636832 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.637521 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:46.637530 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:46.639158 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:46.640886 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.640894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.640897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.641138 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.641376 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.641377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.641377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.641498 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.641770 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:46.641776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:46.641931 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 13:00:46.642048 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:00:46.642204 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260928 13:00:46.643074 24396 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:46.643225 24396 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.643431 24396 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.644513 24396 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:34473 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:00:46.644596 24396 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.644682 24396 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:46.646348 24396 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.646433 24396 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.647135 24396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:00:46.757838 24399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:46.758201 24399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:46.758294 24399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:46.760337 24399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:46.760448 24399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:46.760514 24399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:00:46.762789 24399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.11.84:48977/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:43167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=35835
--tserver_master_addrs=127.11.11.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:46.763074 24399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:46.763437 24399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:46.770480 24405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:46.771001 24399 server_base.cc:1034] running on GCE node
W20260928 13:00:46.771261 24404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:46.771417 24407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:46.771661 24399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:46.771982 24399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:46.773181 24399 hybrid_clock.cc:648] HybridClock initialized: now 1790600446773165 us; error 31 us; skew 500 ppm
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:46.776729 24399 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:00:46.777346 24399 webserver.cc:460] Webserver started at http://127.11.11.67:35835/ using document root <none> and password file <none>
I20260928 13:00:46.777645 24399 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:46.777750 24399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:46.794029 24399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:46.795104 24416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.795446 24399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:46.795559 24399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "20b85d6cf2d248ad9fd45cc7cda5d363"
format_stamp: "Formatted at 2026-09-28 13:00:45 on dist-test-slave-8sjz"
server_key: "7e986beb6321ddc532f35ecc66fb93e9"
server_key_iv: "ce4d6c0796adbfa63f6dfa35f44481d2"
server_key_version: "kuduclusterkey@0"
I20260928 13:00:46.796072 24399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:46.809844 24399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:46.812582 24399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:46.812809 24399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:46.813054 24399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:46.813463 24399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:46.813555 24399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.813623 24399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:46.813673 24399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:46.821118 24399 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:43167
I20260928 13:00:46.821579 24399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:00:46.822103 24529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:43167 every 8 connection(s)
I20260928 13:00:46.822429 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24399
Sep 28 13:00:46 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600446, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:46.832758 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.822860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.11.11.126:34473)
0928 13:00:46.823172 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:46.823173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:46.823392 (+ 219us) server_negotiation.cc:407] Connection header received
0928 13:00:46.823423 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:46.823425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:46.823439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:46.823497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:46.824410 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.825219 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.826348 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.826679 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.826736 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.828704 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:46.828705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:46.828706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:46.828713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:46.831286 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.831755 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.831756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.831757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.831817 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.832107 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.832109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.832109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.832284 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:46.832486 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:46.832545 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 13:00:46.832647 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":283,"thread_start_us":79,"threads_started":1}
I20260928 13:00:46.833582 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.822822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:54119 (local address 127.11.11.126:34473)
0928 13:00:46.822978 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:46.822980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:46.822992 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:00:46.823457 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:46.823458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:46.823467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:46.823497 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:46.824426 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.825219 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.826676 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.826932 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.826976 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.829707 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:46.829708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:46.829709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:46.829715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:46.831474 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.832013 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.832014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.832014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.832104 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:46.832493 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:46.832494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:46.832495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:46.832630 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:46.832675 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:46.833393 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 13:00:46.833482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":130}
I20260928 13:00:46.834106 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:46.822399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:34473 (local address 127.11.11.67:54119)
0928 13:00:46.822596 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:46.822640 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:46.823335 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:46.823696 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:46.823700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:46.823938 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:46.824299 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.824305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.825336 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.825337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.826583 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:46.826586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:46.827085 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:46.827086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:46.827241 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:46.827760 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:46.827770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:46.829581 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:46.831576 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.831584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.831586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.831929 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.832160 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:46.832160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:46.832161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:46.832399 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:46.832730 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:46.832736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:46.833154 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 13:00:46.833751 (+ 597us) client_negotiation.cc:240] Negotiation successful
0928 13:00:46.833930 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":126,"threads_started":1}
I20260928 13:00:46.834599 24530 heartbeater.cc:347] Connected to a master server at 127.11.11.126:34473
I20260928 13:00:46.834700 24530 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.834903 24530 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.835613 24530 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:34473 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:00:46.835675 24530 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.835748 24530 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:46.836133 24530 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:46.836239 24530 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
W20260928 13:00:46.836602 24530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 13:00:47 dist-test-slave-8sjz krb5kdc[23068](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790600447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:47.455662 24262 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:47.455862 24262 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:47.456393 24064 ts_manager.cc:194] Registered new tserver with Master: aa97b7e0e3d347cd86927727690a37ff (127.11.11.65:38357)
I20260928 13:00:47.457275 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:59009
I20260928 13:00:47.647352 24396 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:47.647567 24396 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:47.648015 24064 ts_manager.cc:194] Registered new tserver with Master: dcb12b03bd914fa487b2a5c5704f0fe4 (127.11.11.66:34719)
I20260928 13:00:47.648581 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:53951
Sep 28 13:00:47 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
I20260928 13:00:47.836802 24530 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:47.836966 24530 heartbeater.cc:511] Master 127.11.11.126:34473 requested a full tablet report, sending...
I20260928 13:00:47.837392 24064 ts_manager.cc:194] Registered new tserver with Master: 20b85d6cf2d248ad9fd45cc7cda5d363 (127.11.11.67:43167)
I20260928 13:00:47.837913 24064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:54119
I20260928 13:00:47.839476 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:00:47.853012 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.844379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.11.11.126:34473)
0928 13:00:47.844924 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:47.844927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:47.844935 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:00:47.844960 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:47.844961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:47.844973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:47.845760 (+ 787us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:47.845802 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.846581 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.847638 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.847944 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.848024 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.849716 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:47.849717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:47.849718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:47.849725 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:47.851634 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:47.852071 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:47.852072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:47.852072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:47.852142 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:47.852381 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:47.852382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:47.852382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:47.852549 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:47.852663 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:47.852797 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 13:00:47.852894 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":78,"threads_started":1}
I20260928 13:00:47.855334 24064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:00:47.857795 24064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:00:47.868649 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.861232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38357 (local address 127.0.0.1:33510)
0928 13:00:47.864153 (+ 2921us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:47.864165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:47.864226 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:47.864696 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:47.864697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:47.864698 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:47.864723 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.864724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.865945 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.865946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.867323 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.867326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.868345 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.868346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.868434 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.868450 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 13:00:47.868501 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:00:47.868554 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2901,"thread_start_us":75,"threads_started":1}
I20260928 13:00:47.869114 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.861520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.11.11.65:38357)
0928 13:00:47.861704 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:47.861707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:47.864269 (+ 2562us) server_negotiation.cc:407] Connection header received
0928 13:00:47.864603 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:47.864605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:47.864622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:47.864698 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:47.864772 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.865856 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.867435 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.868222 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.868885 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.868962 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:47.868990 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 13:00:47.869034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260928 13:00:47.869813 24196 tablet_service.cc:1467] Processing CreateTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db (DEFAULT_TABLE table=test-table [id=e9b4fe300a384b609653935cceb56591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:47.870205 24196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f64d4ea6094a8c9611d6f73caed9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:47.871018 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.861122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.11.11.67:43167)
0928 13:00:47.861635 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:47.861640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:47.864222 (+ 2582us) server_negotiation.cc:407] Connection header received
0928 13:00:47.866049 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:47.866053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:47.866068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:47.866119 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:47.866355 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.867422 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.869424 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.870193 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.870273 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.870327 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:47.870893 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 13:00:47.870937 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":50,"threads_started":1}
I20260928 13:00:47.871393 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.861022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:43167 (local address 127.0.0.1:35668)
0928 13:00:47.864142 (+ 3120us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:47.864154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:47.865951 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:47.866223 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:47.866223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:47.866225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:47.866281 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.866282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.867774 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.867775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.869320 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.869323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.870731 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.870731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.870798 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.870802 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:47.871253 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 13:00:47.871317 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"thread_start_us":82,"threads_started":1}
I20260928 13:00:47.872325 24445 tablet_service.cc:1467] Processing CreateTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db (DEFAULT_TABLE table=test-table [id=e9b4fe300a384b609653935cceb56591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:47.872668 24445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f64d4ea6094a8c9611d6f73caed9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:47.876621 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.860999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:34719 (local address 127.0.0.1:49184)
0928 13:00:47.861222 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:47.861272 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:47.861334 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:47.864549 (+ 3215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:47.864553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:47.864556 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:47.864637 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.864638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.874364 (+ 9726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.874365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.875401 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:47.875404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.876392 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.876393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:47.876455 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.876460 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:00:47.876488 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 13:00:47.876531 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260928 13:00:47.877040 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:47.861525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.11.11.66:34719)
0928 13:00:47.864351 (+ 2826us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:47.864356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:47.864367 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:00:47.864406 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:47.864408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:47.864427 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:47.864809 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:47.864890 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.874249 (+ 9359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.875522 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:47.876268 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:47.876806 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:47.876890 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:47.876916 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 13:00:47.876967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2788,"thread_start_us":64,"threads_started":1}
I20260928 13:00:47.877732 24330 tablet_service.cc:1467] Processing CreateTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db (DEFAULT_TABLE table=test-table [id=e9b4fe300a384b609653935cceb56591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:47.878089 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f64d4ea6094a8c9611d6f73caed9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:47.880275 24564 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Bootstrap starting.
I20260928 13:00:47.880503 24563 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Bootstrap starting.
I20260928 13:00:47.881181 24563 tablet_bootstrap.cc:654] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:47.881826 24564 tablet_bootstrap.cc:654] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:47.882165 24564 log.cc:824] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:47.882697 24563 log.cc:824] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:47.883023 24564 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: No bootstrap required, opened a new log
I20260928 13:00:47.883054 24565 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Bootstrap starting.
I20260928 13:00:47.883198 24564 ts_tablet_manager.cc:1397] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:00:47.883733 24565 tablet_bootstrap.cc:654] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:47.884027 24565 log.cc:824] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:47.884816 24563 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: No bootstrap required, opened a new log
I20260928 13:00:47.884889 24563 ts_tablet_manager.cc:1397] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 13:00:47.886438 24564 raft_consensus.cc:348] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.886710 24564 raft_consensus.cc:374] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:47.886791 24564 raft_consensus.cc:729] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b85d6cf2d248ad9fd45cc7cda5d363, State: Initialized, Role: FOLLOWER
I20260928 13:00:47.886993 24564 consensus_queue.cc:260] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.887356 24564 ts_tablet_manager.cc:1428] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 13:00:47.887379 24563 raft_consensus.cc:348] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.887511 24530 heartbeater.cc:503] Master 127.11.11.126:34473 was elected leader, sending a full tablet report...
I20260928 13:00:47.887543 24563 raft_consensus.cc:374] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:47.887581 24563 raft_consensus.cc:729] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa97b7e0e3d347cd86927727690a37ff, State: Initialized, Role: FOLLOWER
I20260928 13:00:47.887732 24563 consensus_queue.cc:260] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.887988 24563 ts_tablet_manager.cc:1428] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:00:47.888160 24262 heartbeater.cc:503] Master 127.11.11.126:34473 was elected leader, sending a full tablet report...
I20260928 13:00:47.889761 24565 tablet_bootstrap.cc:492] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: No bootstrap required, opened a new log
I20260928 13:00:47.889849 24565 ts_tablet_manager.cc:1397] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 13:00:47.892640 24565 raft_consensus.cc:348] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.892817 24565 raft_consensus.cc:374] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:47.892849 24565 raft_consensus.cc:729] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb12b03bd914fa487b2a5c5704f0fe4, State: Initialized, Role: FOLLOWER
I20260928 13:00:47.892964 24565 consensus_queue.cc:260] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:47.893226 24565 ts_tablet_manager.cc:1428] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:00:47.893271 24396 heartbeater.cc:503] Master 127.11.11.126:34473 was elected leader, sending a full tablet report...
W20260928 13:00:47.939522 24263 tablet.cc:2369] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 13:00:48 dist-test-slave-8sjz krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600447, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for HTTP/127.11.11.84@KRBTEST.COM
W20260928 13:00:48.074012 24531 tablet.cc:2369] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:00:48.132318 24397 tablet.cc:2369] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:00:48.206686 24570 raft_consensus.cc:482] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:00:48.206887 24570 raft_consensus.cc:504] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:48.207355 24570 leader_election.cc:288] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcb12b03bd914fa487b2a5c5704f0fe4 (127.11.11.66:34719), 20b85d6cf2d248ad9fd45cc7cda5d363 (127.11.11.67:43167)
I20260928 13:00:48.212481 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.207852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:44887 (local address 127.11.11.67:43167)
0928 13:00:48.207913 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:48.207916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:48.208447 (+ 531us) server_negotiation.cc:407] Connection header received
0928 13:00:48.208556 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:48.208557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:48.208568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:48.208600 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:48.208873 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.209695 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.211363 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.211940 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.212015 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:48.212026 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:48.212345 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 13:00:48.212401 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:00:48.212769 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.207615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:43167 (local address 127.11.11.65:44887)
0928 13:00:48.208372 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:48.208384 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:48.208478 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:48.208671 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:48.208671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:48.208673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:48.208756 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:48.208759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.209822 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.209823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:48.211237 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:48.211241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.212165 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.212166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:48.212240 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:48.212244 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:48.212621 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 13:00:48.212667 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":103,"threads_started":1}
I20260928 13:00:48.213114 24484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f64d4ea6094a8c9611d6f73caed9db" candidate_uuid: "aa97b7e0e3d347cd86927727690a37ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" is_pre_election: true
I20260928 13:00:48.213325 24484 raft_consensus.cc:2454] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa97b7e0e3d347cd86927727690a37ff in term 0.
I20260928 13:00:48.213611 24150 leader_election.cc:302] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b85d6cf2d248ad9fd45cc7cda5d363, aa97b7e0e3d347cd86927727690a37ff; no voters:
I20260928 13:00:48.213768 24570 raft_consensus.cc:2781] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:00:48.213827 24570 raft_consensus.cc:482] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:00:48.213862 24570 raft_consensus.cc:3037] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:48.214835 24570 raft_consensus.cc:504] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:48.214995 24570 leader_election.cc:288] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [CANDIDATE]: Term 1 election: Requested vote from peers dcb12b03bd914fa487b2a5c5704f0fe4 (127.11.11.66:34719), 20b85d6cf2d248ad9fd45cc7cda5d363 (127.11.11.67:43167)
I20260928 13:00:48.215193 24484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f64d4ea6094a8c9611d6f73caed9db" candidate_uuid: "aa97b7e0e3d347cd86927727690a37ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363"
I20260928 13:00:48.215279 24484 raft_consensus.cc:3037] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:48.216223 24484 raft_consensus.cc:2454] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa97b7e0e3d347cd86927727690a37ff in term 1.
I20260928 13:00:48.216784 24150 leader_election.cc:302] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b85d6cf2d248ad9fd45cc7cda5d363, aa97b7e0e3d347cd86927727690a37ff; no voters:
I20260928 13:00:48.216873 24570 raft_consensus.cc:2781] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:48.217027 24570 raft_consensus.cc:686] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 1 LEADER]: Becoming Leader. State: Replica: aa97b7e0e3d347cd86927727690a37ff, State: Running, Role: LEADER
I20260928 13:00:48.217057 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.208122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47207 (local address 127.11.11.66:34719)
0928 13:00:48.208186 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:48.208188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:48.213699 (+ 5511us) server_negotiation.cc:407] Connection header received
0928 13:00:48.213828 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:48.213829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:48.213841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:48.213873 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:48.214085 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.214765 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.216096 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.216626 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.216673 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:48.216683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:48.216926 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 13:00:48.216977 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:00:48.217126 24570 consensus_queue.cc:237] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } }
I20260928 13:00:48.217299 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.208026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:34719 (local address 127.11.11.65:47207)
0928 13:00:48.213623 (+ 5597us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:48.213636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:48.213732 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:48.213941 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:48.213942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:48.213944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:48.214009 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:48.214010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.214884 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.214885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:48.215962 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:48.215964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.216743 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.216744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:48.216833 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:48.216837 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:48.217161 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 13:00:48.217201 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5575,"thread_start_us":103,"threads_started":1}
I20260928 13:00:48.217665 24350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f64d4ea6094a8c9611d6f73caed9db" candidate_uuid: "aa97b7e0e3d347cd86927727690a37ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" is_pre_election: true
I20260928 13:00:48.217849 24350 raft_consensus.cc:2454] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa97b7e0e3d347cd86927727690a37ff in term 0.
I20260928 13:00:48.218006 24349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f64d4ea6094a8c9611d6f73caed9db" candidate_uuid: "aa97b7e0e3d347cd86927727690a37ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4"
I20260928 13:00:48.218086 24349 raft_consensus.cc:3037] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:48.218557 24064 catalog_manager.cc:5495] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff reported cstate change: term changed from 0 to 1, leader changed from <none> to aa97b7e0e3d347cd86927727690a37ff (127.11.11.65). New cstate: current_term: 1 leader_uuid: "aa97b7e0e3d347cd86927727690a37ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa97b7e0e3d347cd86927727690a37ff" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38357 } health_report { overall_health: HEALTHY } } }
I20260928 13:00:48.218966 24349 raft_consensus.cc:2454] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa97b7e0e3d347cd86927727690a37ff in term 1.
I20260928 13:00:48.309253 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.306351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.11.11.65:38357)
0928 13:00:48.306401 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:48.306403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:48.306415 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:00:48.306491 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:48.306491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:48.306515 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:48.306604 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:00:48.306725 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.307495 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.308304 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:48.308608 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:48.308696 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:48.308838 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:00:48.308966 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:00:48.309062 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:48.309121 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 13:00:48.309146 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 13:00:48.311826 24349 raft_consensus.cc:1261] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 1 FOLLOWER]: Refusing update from remote peer aa97b7e0e3d347cd86927727690a37ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:48.311837 24484 raft_consensus.cc:1261] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 1 FOLLOWER]: Refusing update from remote peer aa97b7e0e3d347cd86927727690a37ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:48.312249 24570 consensus_queue.cc:1035] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb12b03bd914fa487b2a5c5704f0fe4" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:48.312462 24576 consensus_queue.cc:1035] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b85d6cf2d248ad9fd45cc7cda5d363" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:48.315450 24586 mvcc.cc:205] Tried to move back new op lower bound from 7334299436281917440 to 7334299435897950208. Current Snapshot: MvccSnapshot[applied={T|T < 7334299436281917440}]
I20260928 13:00:48.317867 24064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:48.317976 24064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:48.318723 24064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d843e0b07f5a4e589a08a2581061a759: Sending DeleteTablet for 3 replicas of tablet 97f64d4ea6094a8c9611d6f73caed9db
I20260928 13:00:48.319077 24330 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:48 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:49184
I20260928 13:00:48.319180 24196 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:48 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:33510
I20260928 13:00:48.319252 24445 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f64d4ea6094a8c9611d6f73caed9db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:48 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:35668
I20260928 13:00:48.319470 24597 tablet_replica.cc:330] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff: stopping tablet replica
I20260928 13:00:48.319566 24597 raft_consensus.cc:2229] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:00:48.319662 24597 raft_consensus.cc:2258] T 97f64d4ea6094a8c9611d6f73caed9db P aa97b7e0e3d347cd86927727690a37ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:48.319864 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24102
I20260928 13:00:48.319784 24598 tablet_replica.cc:330] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: stopping tablet replica
I20260928 13:00:48.320577 24598 raft_consensus.cc:2229] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:48.320662 24596 tablet_replica.cc:330] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: stopping tablet replica
I20260928 13:00:48.320695 24598 raft_consensus.cc:2258] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:48.320753 24596 raft_consensus.cc:2229] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:48.320863 24596 raft_consensus.cc:2258] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:48.321159 24598 ts_tablet_manager.cc:1905] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:48.322448 24596 ts_tablet_manager.cc:1905] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:48.323024 24598 ts_tablet_manager.cc:1918] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:48.323099 24598 log.cc:1196] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/97f64d4ea6094a8c9611d6f73caed9db
I20260928 13:00:48.323421 24598 ts_tablet_manager.cc:1939] T 97f64d4ea6094a8c9611d6f73caed9db P 20b85d6cf2d248ad9fd45cc7cda5d363: Deleting consensus metadata
I20260928 13:00:48.323954 24049 catalog_manager.cc:4841] TS 20b85d6cf2d248ad9fd45cc7cda5d363 (127.11.11.67:43167): tablet 97f64d4ea6094a8c9611d6f73caed9db (table test-table [id=e9b4fe300a384b609653935cceb56591]) successfully deleted
I20260928 13:00:48.324257 24596 ts_tablet_manager.cc:1918] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:48.324332 24596 log.cc:1196] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/97f64d4ea6094a8c9611d6f73caed9db
I20260928 13:00:48.324608 24596 ts_tablet_manager.cc:1939] T 97f64d4ea6094a8c9611d6f73caed9db P dcb12b03bd914fa487b2a5c5704f0fe4: Deleting consensus metadata
I20260928 13:00:48.325074 24051 catalog_manager.cc:4841] TS dcb12b03bd914fa487b2a5c5704f0fe4 (127.11.11.66:34719): tablet 97f64d4ea6094a8c9611d6f73caed9db (table test-table [id=e9b4fe300a384b609653935cceb56591]) successfully deleted
W20260928 13:00:48.325574 24051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:38357 (error 108)
I20260928 13:00:48.326150 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:48.326004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38357 (local address 127.0.0.1:33538)
0928 13:00:48.326069 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:48.326119 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 13:00:48.326279 24051 catalog_manager.cc:4584] TS aa97b7e0e3d347cd86927727690a37ff (127.11.11.65:38357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97f64d4ea6094a8c9611d6f73caed9db: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38357: connect: Connection refused (error 111)
I20260928 13:00:48.326323 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24266
I20260928 13:00:48.331358 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24399
I20260928 13:00:48.335894 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24031
I20260928 13:00:48.341764 11309 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 13:00:48.686939 11309 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 13:00:48.687040 11309 mini_ranger.cc:66] Stopping Ranger...
I20260928 13:00:49.032960 11309 mini_ranger.cc:68] Stopped Ranger
2026-09-28 13:00:49.033 UTC [23098] LOG: received smart shutdown request
2026-09-28 13:00:49.037 UTC [23098] LOG: background worker "logical replication launcher" (PID 23107) exited with exit code 1
2026-09-28 13:00:49.037 UTC [23102] LOG: shutting down
2026-09-28 13:00:49.227 UTC [23098] LOG: database system is shut down
2026-09-28T13:00:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49859 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:00:49 dist-test-slave-8sjz krb5kdc[24608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:00:49 dist-test-slave-8sjz krb5kdc[24608](info): set up 2 sockets
Sep 28 13:00:49 dist-test-slave-8sjz krb5kdc[24608](info): commencing operation
krb5kdc: starting...
W20260928 13:00:50.385495 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:00:50.510741 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:32827
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:50.592124 24621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:50.592399 24621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:50.592442 24621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:50.593676 24621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:50.593755 24621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:50.593778 24621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:50.593797 24621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:50.593814 24621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:50.595391 24621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:50.595685 24621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:50.595927 24621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:50.600811 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.600811 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.600843 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:50.601308 24621 server_base.cc:1034] running on GCE node
I20260928 13:00:50.601527 24621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:50.601779 24621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:50.602919 24621 hybrid_clock.cc:648] HybridClock initialized: now 1790600450602907 us; error 33 us; skew 500 ppm
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:50.605491 24621 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:50.605957 24621 webserver.cc:460] Webserver started at http://127.11.11.126:41385/ using document root <none> and password file <none>
I20260928 13:00:50.606215 24621 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:50.606261 24621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:50.606403 24621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:50.607506 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "4e10a92b43c24ebf88fc074ff5d416b1"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "368d22714de6053a43f2c6eee8a3622b"
server_key_iv: "c7f16187c878a20810d6bb53513e0ba5"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.607869 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "4e10a92b43c24ebf88fc074ff5d416b1"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "368d22714de6053a43f2c6eee8a3622b"
server_key_iv: "c7f16187c878a20810d6bb53513e0ba5"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.609254 24621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:50.610088 24636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:50.610343 24621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:50.610469 24621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "4e10a92b43c24ebf88fc074ff5d416b1"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "368d22714de6053a43f2c6eee8a3622b"
server_key_iv: "c7f16187c878a20810d6bb53513e0ba5"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.610545 24621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:50.622480 24621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:50.624449 24621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:50.624610 24621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:50.628603 24621 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:32827
I20260928 13:00:50.628643 24688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:32827 every 8 connection(s)
I20260928 13:00:50.629004 24621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:50.629868 24689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:50.631449 24689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.635756 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24621
I20260928 13:00:50.635849 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:00:50.636099 11309 external_mini_cluster.cc:1428] Setting key 1ca7085b67cc2f1069d8ecc4c2894801
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:50.641860 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.632272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:32827 (local address 127.0.0.1:39164)
0928 13:00:50.632486 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.632525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.632985 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.633302 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.633304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.633561 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.633718 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.633722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.634751 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.634752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.635464 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.635466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.636078 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.636080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.636149 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.636713 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.636722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.638187 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.640345 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.640351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.640353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.640548 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.640838 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.640839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.640839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.640923 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.641417 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.641421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.641478 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.641550 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.641679 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"spinlock_wait_cycles":8576,"thread_start_us":101,"threads_started":1}
I20260928 13:00:50.641954 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.632272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.11.11.126:32827)
0928 13:00:50.632543 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.632547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.632602 (+ 55us) server_negotiation.cc:407] Connection header received
0928 13:00:50.633149 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.633152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.633175 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.633266 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.633844 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.634655 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.635542 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.635926 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.636094 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.638275 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.638282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.638283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.638292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.640258 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.640641 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.640645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.640645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.640760 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.640993 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.640994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.640995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.641328 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.641442 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.641612 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.641772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260928 13:00:50.642753 24689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.643149 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.643018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39896)
0928 13:00:50.643070 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.643117 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:00:50.643322 24639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
I20260928 13:00:50.643781 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.643687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39898)
0928 13:00:50.643733 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.643757 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 13:00:50.643914 24689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:45999: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.11.11.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
I20260928 13:00:50.663493 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:45999
--webserver_interface=127.11.11.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:50.693199 24689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } attempt: 1
I20260928 13:00:50.693765 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.693611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39908)
0928 13:00:50.693681 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.693730 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 13:00:50.694195 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.694105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39918)
0928 13:00:50.694151 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.694177 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 13:00:50.694307 24689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:45999: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
W20260928 13:00:50.746285 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:50.746512 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:50.746536 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:50.747757 24696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:50.747805 24696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:50.747820 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:50.747833 24696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:50.747845 24696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:50.749441 24696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.11.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:50.749717 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:50.749982 24696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:50.754909 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.754945 24702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.754909 24701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:50.755120 24696 server_base.cc:1034] running on GCE node
I20260928 13:00:50.755455 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:50.755718 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:50.756872 24696 hybrid_clock.cc:648] HybridClock initialized: now 1790600450756857 us; error 31 us; skew 500 ppm
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:50.759544 24696 init.cc:377] Logged in from keytab as kudu/127.11.11.125@KRBTEST.COM (short username kudu)
I20260928 13:00:50.760022 24696 webserver.cc:460] Webserver started at http://127.11.11.125:39199/ using document root <none> and password file <none>
I20260928 13:00:50.760314 24696 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:50.760388 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:50.760551 24696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:50.761655 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data/instance:
uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "0d516a9dba53577abac8ed885b731963"
server_key_iv: "82327fa3c122dbd4d4de502470f0a6f6"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.762053 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal/instance:
uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "0d516a9dba53577abac8ed885b731963"
server_key_iv: "82327fa3c122dbd4d4de502470f0a6f6"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.762462 24689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } attempt: 2
I20260928 13:00:50.762866 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.762732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39920)
0928 13:00:50.762797 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.762832 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 13:00:50.763279 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.763188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39922)
0928 13:00:50.763236 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.763261 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:00:50.763391 24689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:45999: Network error: Client connection negotiation failed: client connection to 127.11.11.125:45999: connect: Connection refused (error 111)
I20260928 13:00:50.763803 24696 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 13:00:50.764636 24711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:50.764832 24696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:50.764921 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "0d516a9dba53577abac8ed885b731963"
server_key_iv: "82327fa3c122dbd4d4de502470f0a6f6"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.764995 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:50.782702 24696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:50.785251 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:50.785434 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:50.789628 24696 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.125:45999
I20260928 13:00:50.789700 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.125:45999 every 8 connection(s)
I20260928 13:00:50.790062 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
I20260928 13:00:50.791011 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:50.792608 24764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:50.798712 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24696
I20260928 13:00:50.798799 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-1/wal/instance
I20260928 13:00:50.799019 11309 external_mini_cluster.cc:1428] Setting key 277b40b790797d5090e2c7a271593349
I20260928 13:00:50.801694 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.793360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.11.11.126:32827)
0928 13:00:50.793447 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.793449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.793638 (+ 189us) server_negotiation.cc:407] Connection header received
0928 13:00:50.794077 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.794079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.794088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.794137 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.794728 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.795212 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.796246 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.796446 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.796502 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.798641 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.798643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.798644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.798653 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.800228 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.800679 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.800681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.800682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.800742 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.801002 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.801003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.801003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.801124 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.801201 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.801485 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.801601 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 13:00:50.801893 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.793307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:32827 (local address 127.0.0.1:39172)
0928 13:00:50.793515 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.793554 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.793995 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.794202 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.794205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.794459 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.794658 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.794663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.795290 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.795290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.796170 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.796172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.796517 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.796518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.796619 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.797154 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.797163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.798538 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.800328 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.800336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.800338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.800583 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.800816 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.800817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.800817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.800934 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.801193 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.801199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.801346 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.801464 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.801643 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260928 13:00:50.802466 24764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.11.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.124@KRBTEST.COM" created.
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:00:50.813323 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.802763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39926)
0928 13:00:50.802808 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.802818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.802851 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.803173 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.803174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.803328 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.803383 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.803384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.804333 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.804334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.804831 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.804833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.805306 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.805307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.805374 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.805810 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.805815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.809779 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.811973 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.811974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.811975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.812192 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.812422 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.812422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.812423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.812478 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.812848 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.812850 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.812921 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.813033 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.813145 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:00:50.813345 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.802851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.11.11.125:45999)
0928 13:00:50.803024 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.803027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.803043 (+ 16us) server_negotiation.cc:407] Connection header received
0928 13:00:50.803059 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.803060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.803071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.803171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.803462 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.804232 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.804927 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.805222 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.805298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.809898 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.809910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.809915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.809922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.811875 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.812286 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.812290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.812290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.812354 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.812552 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.812553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.812553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.812747 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.812847 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.813103 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.813214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260928 13:00:50.814071 24764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.814468 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.814316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52790)
0928 13:00:50.814382 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.814433 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:00:50.814631 24714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
I20260928 13:00:50.815083 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.814993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52800)
0928 13:00:50.815042 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.815067 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 13:00:50.815207 24764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
I20260928 13:00:50.827175 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:36181
--webserver_interface=127.11.11.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:50.854640 24689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } attempt: 3
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:00:50.861928 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.855211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.11.11.125:45999)
0928 13:00:50.855276 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.855278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.855289 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:00:50.855382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.855383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.855394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.855454 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.855798 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.856375 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.857119 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.857336 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.857421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.859398 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.859400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.859400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.859407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.860883 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.861269 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.861269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.861270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.861318 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.861512 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.861512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.861513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.861613 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.861659 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.861797 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.861859 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:00:50.861982 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.855113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39928)
0928 13:00:50.855192 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.855211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.855297 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.855476 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.855477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.855645 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.855708 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.855710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.856458 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.856459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.856988 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.856990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.857418 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.857419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.857505 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.857988 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.857993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.859301 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.860967 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.860968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.860968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.861193 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.861384 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.861385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.861385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.861454 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.861662 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.861663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.861715 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.861779 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.861884 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 13:00:50.862339 24689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.862735 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.862628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52808)
0928 13:00:50.862683 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.862713 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:00:50.863114 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.863022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52816)
0928 13:00:50.863068 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.863093 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 13:00:50.863225 24689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
I20260928 13:00:50.864467 24764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } attempt: 1
I20260928 13:00:50.865020 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.864860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52828)
0928 13:00:50.864925 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.864986 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 13:00:50.865350 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.865255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52830)
0928 13:00:50.865301 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.865326 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:00:50.865458 24764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
I20260928 13:00:50.894418 24689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } attempt: 1
I20260928 13:00:50.895022 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.894871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52846)
0928 13:00:50.894946 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.894992 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 13:00:50.895437 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.895340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52856)
0928 13:00:50.895394 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.895420 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 13:00:50.895567 24689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
W20260928 13:00:50.912495 24771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:50.912755 24771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:50.912782 24771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:50.914057 24771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:50.914103 24771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:50.914117 24771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:50.914130 24771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:50.914142 24771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:50.915752 24771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:32827,127.11.11.125:45999,127.11.11.124:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.11.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:50.915968 24771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:50.916225 24771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:50.921103 24779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.921123 24776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:50.921123 24777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:50.921437 24771 server_base.cc:1034] running on GCE node
I20260928 13:00:50.921705 24771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:50.922009 24771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:50.923205 24771 hybrid_clock.cc:648] HybridClock initialized: now 1790600450923187 us; error 49 us; skew 500 ppm
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:50.925953 24771 init.cc:377] Logged in from keytab as kudu/127.11.11.124@KRBTEST.COM (short username kudu)
I20260928 13:00:50.926406 24771 webserver.cc:460] Webserver started at http://127.11.11.124:41993/ using document root <none> and password file <none>
I20260928 13:00:50.926666 24771 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:50.926735 24771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:50.926887 24771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:50.927976 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data/instance:
uuid: "18f99b099bb740ab9b737ce8b740e3db"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "4a3a7ceb2b5e29bb5fbbc367afcf6964"
server_key_iv: "e1a8a6aa4a0fdd5c86cdeb3e19f06509"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.928419 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal/instance:
uuid: "18f99b099bb740ab9b737ce8b740e3db"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "4a3a7ceb2b5e29bb5fbbc367afcf6964"
server_key_iv: "e1a8a6aa4a0fdd5c86cdeb3e19f06509"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.929791 24771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:50.930565 24786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:50.930745 24771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:50.930852 24771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "18f99b099bb740ab9b737ce8b740e3db"
format_stamp: "Formatted at 2026-09-28 13:00:50 on dist-test-slave-8sjz"
server_key: "4a3a7ceb2b5e29bb5fbbc367afcf6964"
server_key_iv: "e1a8a6aa4a0fdd5c86cdeb3e19f06509"
server_key_version: "encryptionkey@0"
I20260928 13:00:50.930946 24771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:50.933593 24764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } attempt: 2
I20260928 13:00:50.934003 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.933880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52868)
0928 13:00:50.933937 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.933974 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:00:50.934376 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.934239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52870)
0928 13:00:50.934295 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.934336 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:00:50.934496 24764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.124:36181: connect: Connection refused (error 111)
I20260928 13:00:50.950299 24771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:50.953001 24771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:50.953178 24771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:50.957392 24771 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.124:36181
I20260928 13:00:50.957434 24838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.124:36181 every 8 connection(s)
I20260928 13:00:50.957834 24771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
I20260928 13:00:50.958691 24839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:50.960363 24839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.962222 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24771
I20260928 13:00:50.962322 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600290352246-11309-0/minicluster-data/master-2/wal/instance
I20260928 13:00:50.962528 11309 external_mini_cluster.cc:1428] Setting key 601056c1017403917591e94d85e5434e
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:00:50.970736 24689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } attempt: 2
I20260928 13:00:50.971002 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.961140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.11.11.126:32827)
0928 13:00:50.961213 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.961216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.961420 (+ 204us) server_negotiation.cc:407] Connection header received
0928 13:00:50.961863 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.961865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.961879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.961945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.962867 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.963453 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.964721 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.964913 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.964968 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.967504 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.967506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.967506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.967513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.969214 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.969971 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.969972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.969973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.970038 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.970293 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.970294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.970294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.970419 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.970497 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.970772 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.970877 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:00:50.971494 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.961126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:32827 (local address 127.0.0.1:39186)
0928 13:00:50.961294 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.961331 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.961772 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.962051 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.962056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.962446 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.962754 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.962759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.963528 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.963529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.964635 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.964637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.964992 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.964993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.965149 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.965731 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.965740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.967421 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.969301 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.969309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.969311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.969888 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.970113 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.970113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.970114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.970223 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.970489 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.970496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.970636 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.970750 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.971233 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 13:00:50.972075 24839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.972621 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.962892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.11.11.124:36181)
0928 13:00:50.963042 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.963046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.963649 (+ 603us) server_negotiation.cc:407] Connection header received
0928 13:00:50.963670 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.963672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.963694 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.963781 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.964391 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.965343 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.966152 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.966541 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.966600 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.969205 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.969214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.969214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.969221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.970970 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.971387 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.971391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.971391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.971448 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.971685 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.971686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.971686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.971855 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.972000 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.972261 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.972470 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:50.980077 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.971002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52880)
0928 13:00:50.971054 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.971064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.971103 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.971514 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.971516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.971677 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.971723 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.971724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.973019 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.973020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.973761 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.973764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.974232 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.974233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.974301 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.974679 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.974684 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.976138 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.978914 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.978915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.978916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.979141 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.979365 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.979366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.979439 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.979686 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.979687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.979736 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.979806 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.979927 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:00:50.980151 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.972398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.11.11.125:45999)
0928 13:00:50.972441 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.972443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.972453 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:00:50.972526 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.972527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.972536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.972592 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.972893 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.973613 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.974681 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.974872 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.974956 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.977441 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.977442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.977442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.977450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.978872 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979242 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.979243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.979244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.979301 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979546 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.979547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.979548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.979682 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.979727 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.979960 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.980064 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:00:50.980247 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.972325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:45999 (local address 127.0.0.1:39930)
0928 13:00:50.972395 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.972406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.972452 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.972603 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.972604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.972761 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.972810 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.972811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.973689 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.973690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.974381 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.974383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.974952 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.974953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.975025 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.975872 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.975877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.977338 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.978996 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.978997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.978997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.979158 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.979404 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.979405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.979465 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.979776 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.979777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.979822 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.979927 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.980025 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 13:00:50.980450 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.971051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.11.11.124:36181)
0928 13:00:50.971382 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.971384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.971392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:00:50.971411 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.971412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.971421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.971493 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.971802 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.972926 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.973854 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.974145 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.974471 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.976222 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.976223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.976223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.976230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.978808 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979231 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.979232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.979233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.979292 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.979506 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.979507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.979507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.979608 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.979691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.980220 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.980317 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":98,"threads_started":1}
I20260928 13:00:50.980859 24839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:00:50.982978 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.974008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.11.11.126:32827)
0928 13:00:50.974040 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.974041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.974048 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:00:50.974067 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.974068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.974076 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.974130 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.974755 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.975387 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.976176 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.976475 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.976548 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.979524 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.979525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.979525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.979532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.981378 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.981734 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.981735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.981735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.981794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.981986 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.981987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.981987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.982107 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.982775 (+ 668us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.982820 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.982897 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 13:00:50.983038 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: Bootstrap starting.
I20260928 13:00:50.983809 24689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:50.984200 24689 log.cc:824] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:50.985112 24689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: No bootstrap required, opened a new log
I20260928 13:00:50.987269 24689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:50.987422 24689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:50.987475 24689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e10a92b43c24ebf88fc074ff5d416b1, State: Initialized, Role: FOLLOWER
I20260928 13:00:50.987607 24689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:50.987963 24689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:50.987907 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:50.988066 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
W20260928 13:00:50.989877 24857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:00:50.989987 24857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 13:00:50 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:00:50.992802 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.983616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.11.11.125:45999)
0928 13:00:50.983655 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.983657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.983665 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:00:50.983688 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.983688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.983699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.983769 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.984359 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.985110 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.985942 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.986146 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.986232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.988831 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.988833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.988833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.988841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.990888 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.991281 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.991283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.991283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.991348 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.991569 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.991570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.991570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.991708 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.992550 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.992596 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.992676 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:00:50.993927 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.981110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52892)
0928 13:00:50.981509 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:50.981517 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:50.981582 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:50.984571 (+ 2989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:50.984571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:50.984724 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:50.984762 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.984764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.985628 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.985629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.986367 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:50.986370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.988843 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.988844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:50.988915 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.989520 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:50.989525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:50.990820 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:50.992841 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.992842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.992842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.993059 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.993259 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:50.993259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:50.993260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:50.993311 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:50.993605 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:50.993605 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:50.993644 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 13:00:50.993713 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:00:50.993811 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386}
I20260928 13:00:50.993944 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.981173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.11.11.124:36181)
0928 13:00:50.981205 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.981207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.984465 (+ 3258us) server_negotiation.cc:407] Connection header received
0928 13:00:50.984486 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.984486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.984496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.984569 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.984847 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.985545 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.986463 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:50.986769 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:50.986849 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:50.990925 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:50.990926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:50.990927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:50.990933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:50.992685 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.993121 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.993122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.993123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.993172 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:50.993388 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:50.993389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:50.993389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:50.993536 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:50.993613 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:50.993737 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 13:00:50.993860 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":16}
I20260928 13:00:50.991999 24621 master_runner.cc:186] Error getting master registration for 127.11.11.125:45999: OK, instance_id { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" instance_seqno: 1790600450788403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:00:50.994868 24621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:00:50.996483 24839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: Bootstrap starting.
I20260928 13:00:50.996858 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:00:50.997277 24839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:50.997592 24839 log.cc:824] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:50.998418 24839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: No bootstrap required, opened a new log
I20260928 13:00:50.999828 24839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:50.999987 24839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:51.000036 24839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f99b099bb740ab9b737ce8b740e3db, State: Initialized, Role: FOLLOWER
I20260928 13:00:51.000196 24839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.000438 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.000501 24839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:51.000504 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:00:51.002243 24876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:00:51.002290 24876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:00:51.007004 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:50.998330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.11.11.126:32827)
0928 13:00:50.998372 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:50.998375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:50.998611 (+ 236us) server_negotiation.cc:407] Connection header received
0928 13:00:50.998710 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:50.998711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:50.998721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:50.998752 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:50.999515 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.000295 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.001418 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.001707 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.001750 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:51.003937 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:51.003939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:51.003939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:51.003947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:51.005754 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.006146 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.006147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.006147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.006203 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.006419 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.006420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.006421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.006553 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:51.006628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:51.006838 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 13:00:51.006920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:00:51.004089 24771 master_runner.cc:186] Error getting master registration for 127.11.11.125:45999: OK, instance_id { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" instance_seqno: 1790600450788403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:00:51.007303 24771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:00:51.010829 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:51.002911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.11.11.124:36181)
0928 13:00:51.002950 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:51.002951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:51.003049 (+ 98us) server_negotiation.cc:407] Connection header received
0928 13:00:51.003156 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:51.003157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:51.003167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:51.003197 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:51.003805 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.004787 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.005953 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.006474 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.006517 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:51.007901 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:51.007902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:51.007902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:51.007909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:51.009610 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.010021 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.010022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.010022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.010084 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.010305 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.010306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.010306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.010449 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:51.010554 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:51.010659 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 13:00:51.010737 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:00:51.015022 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:51.002194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.11.11.125:45999)
0928 13:00:51.002242 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:51.002244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:51.002771 (+ 527us) server_negotiation.cc:407] Connection header received
0928 13:00:51.002877 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:51.002879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:51.002891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:51.002924 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:51.003717 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.004477 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.005460 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.005751 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.005788 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:51.011707 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:51.011709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:51.011709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:51.011717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:51.013426 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.013811 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.013812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.013812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.013874 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.014102 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.014102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.014103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.014254 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:51.014364 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:51.014836 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 13:00:51.014923 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:00:51.025691 24764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } attempt: 3
Sep 28 13:00:51 dist-test-slave-8sjz krb5kdc[24608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600450, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:00:51.037707 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:51.026043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:36181 (local address 127.0.0.1:52914)
0928 13:00:51.026110 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:51.026122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:51.026225 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:51.026749 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:51.026751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:51.027021 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:51.027105 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:51.027108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.028505 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.028506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:51.029535 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:51.029538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.030279 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.030280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:51.030353 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:51.032279 (+ 1926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:51.032286 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:51.034159 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:51.036499 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:51.036501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:51.036501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:51.036732 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:51.036978 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:51.036979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:51.036980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:51.037050 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:51.037358 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:51.037359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:51.037414 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 13:00:51.037490 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:00:51.037598 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 13:00:51.037724 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:51.026493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.11.11.124:36181)
0928 13:00:51.026583 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:51.026585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:51.026592 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:00:51.026614 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:51.026615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:51.026649 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:51.026704 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:51.027268 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.028374 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.029698 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:51.030108 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:51.030218 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:51.034237 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:51.034238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:51.034238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:51.034246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:51.036410 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.036829 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.036831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.036831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.036894 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:51.037132 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:51.037133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:51.037133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:51.037282 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:51.037352 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:51.037533 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 13:00:51.037617 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 13:00:51.039646 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: Bootstrap starting.
I20260928 13:00:51.040459 24764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:51.040773 24764 log.cc:824] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:51.041690 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: No bootstrap required, opened a new log
I20260928 13:00:51.043776 24764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.043922 24764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:51.043959 24764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d96eb239da14a7cb5c3cb77a0af8b5d, State: Initialized, Role: FOLLOWER
I20260928 13:00:51.044191 24764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.044488 24881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.044539 24764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:51.044625 24881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:00:51.046670 24892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:00:51.046725 24892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:00:51.048503 24696 master_runner.cc:263] No leader master found from master 0d96eb239da14a7cb5c3cb77a0af8b5d
I20260928 13:00:51.048549 24696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:00:51.106056 24846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:00:51.106201 24846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.106494 24846 leader_election.cc:288] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d96eb239da14a7cb5c3cb77a0af8b5d (127.11.11.125:45999), 18f99b099bb740ab9b737ce8b740e3db (127.11.11.124:36181)
I20260928 13:00:51.106995 24813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f99b099bb740ab9b737ce8b740e3db" is_pre_election: true
I20260928 13:00:51.107172 24813 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e10a92b43c24ebf88fc074ff5d416b1 in term 0.
I20260928 13:00:51.107524 24639 leader_election.cc:302] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f99b099bb740ab9b737ce8b740e3db, 4e10a92b43c24ebf88fc074ff5d416b1; no voters:
I20260928 13:00:51.107970 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" is_pre_election: true
I20260928 13:00:51.108143 24738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e10a92b43c24ebf88fc074ff5d416b1 in term 0.
I20260928 13:00:51.108585 24846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:00:51.108654 24846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:00:51.108680 24846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:51.109705 24846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.109839 24846 leader_election.cc:288] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [CANDIDATE]: Term 1 election: Requested vote from peers 0d96eb239da14a7cb5c3cb77a0af8b5d (127.11.11.125:45999), 18f99b099bb740ab9b737ce8b740e3db (127.11.11.124:36181)
I20260928 13:00:51.110159 24813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f99b099bb740ab9b737ce8b740e3db"
I20260928 13:00:51.110250 24813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:51.111208 24813 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e10a92b43c24ebf88fc074ff5d416b1 in term 1.
I20260928 13:00:51.111475 24639 leader_election.cc:302] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f99b099bb740ab9b737ce8b740e3db, 4e10a92b43c24ebf88fc074ff5d416b1; no voters:
I20260928 13:00:51.111716 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d"
I20260928 13:00:51.111804 24738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:51.113580 24738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e10a92b43c24ebf88fc074ff5d416b1 in term 1.
I20260928 13:00:51.114508 24846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:51.114666 24846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [term 1 LEADER]: Becoming Leader. State: Replica: 4e10a92b43c24ebf88fc074ff5d416b1, State: Running, Role: LEADER
I20260928 13:00:51.114758 24846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } }
I20260928 13:00:51.115164 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e10a92b43c24ebf88fc074ff5d416b1. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.115227 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:51.115562 24896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:51.115810 24896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:51.118170 24813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [term 1 FOLLOWER]: Refusing update from remote peer 4e10a92b43c24ebf88fc074ff5d416b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:51.118582 24846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:51.120002 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e10a92b43c24ebf88fc074ff5d416b1. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.120131 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:00:51.120646 24738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [term 1 FOLLOWER]: Refusing update from remote peer 4e10a92b43c24ebf88fc074ff5d416b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:51.120944 24846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:51.122242 24881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e10a92b43c24ebf88fc074ff5d416b1. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.122334 24881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:00:51.123553 24894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.123660 24894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:51.124145 24881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.124226 24881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:00:51.124475 24897 mvcc.cc:205] Tried to move back new op lower bound from 7334299447777579008 to 7334299447766749184. Current Snapshot: MvccSnapshot[applied={T|T < 7334299447777579008}]
I20260928 13:00:51.125041 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.125130 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:51.124997 24900 mvcc.cc:205] Tried to move back new op lower bound from 7334299447777579008 to 7334299447766749184. Current Snapshot: MvccSnapshot[applied={T|T < 7334299447777579008}]
I20260928 13:00:51.125567 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e10a92b43c24ebf88fc074ff5d416b1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 32827 } } peers { permanent_uuid: "0d96eb239da14a7cb5c3cb77a0af8b5d" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 45999 } } peers { permanent_uuid: "18f99b099bb740ab9b737ce8b740e3db" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 36181 } } }
I20260928 13:00:51.125662 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:00:51.125746 24898 mvcc.cc:205] Tried to move back new op lower bound from 7334299447777579008 to 7334299447766749184. Current Snapshot: MvccSnapshot[applied={T|T < 7334299447777579008}]
I20260928 13:00:51.126129 24896 catalog_manager.cc:1347] Generated new cluster ID: 0782ad13ae364121b67b133c9d97e0c2
I20260928 13:00:51.126183 24896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:51.141503 24896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:00:51.142316 24896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:51.155434 24896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e10a92b43c24ebf88fc074ff5d416b1: Generated new TSK 0
I20260928 13:00:51.155743 24896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:00:52.002729 24876 catalog_manager.cc:1259] Loaded cluster ID: 0782ad13ae364121b67b133c9d97e0c2
I20260928 13:00:52.002810 24876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: loading cluster ID for follower catalog manager: success
I20260928 13:00:52.003664 24876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: acquiring CA information for follower catalog manager: success
I20260928 13:00:52.003993 24876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18f99b099bb740ab9b737ce8b740e3db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:00:52.047158 24892 catalog_manager.cc:1259] Loaded cluster ID: 0782ad13ae364121b67b133c9d97e0c2
I20260928 13:00:52.047240 24892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: loading cluster ID for follower catalog manager: success
I20260928 13:00:52.048128 24892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: acquiring CA information for follower catalog manager: success
I20260928 13:00:52.048517 24892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d96eb239da14a7cb5c3cb77a0af8b5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:00:52.209266 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24621
I20260928 13:00:52.214931 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24696
I20260928 13:00:52.220386 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24771
2026-09-28T13:00:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2879 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:00:52 dist-test-slave-8sjz krb5kdc[24914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:00:52 dist-test-slave-8sjz krb5kdc[24914](info): set up 2 sockets
Sep 28 13:00:52 dist-test-slave-8sjz krb5kdc[24914](info): commencing operation
krb5kdc: starting...
W20260928 13:00:53.273111 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:00:53.400321 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38369
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:53.481992 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:53.482257 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:53.482300 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:53.483521 24927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:53.483593 24927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:53.483620 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:53.483642 24927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:53.483669 24927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:53.485256 24927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:38369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:53.485548 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:53.485795 24927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:53.491150 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:53.491155 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:53.491155 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:53.491396 24927 server_base.cc:1034] running on GCE node
I20260928 13:00:53.491783 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:53.492174 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:53.493342 24927 hybrid_clock.cc:648] HybridClock initialized: now 1790600453493327 us; error 33 us; skew 500 ppm
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[24914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600453, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:53.496104 24927 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:53.496596 24927 webserver.cc:460] Webserver started at http://127.11.11.126:35313/ using document root <none> and password file <none>
I20260928 13:00:53.496834 24927 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:53.496901 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:53.497058 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:53.498143 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "d7e5820c82904c47b7d3fa91256082c1"
format_stamp: "Formatted at 2026-09-28 13:00:53 on dist-test-slave-8sjz"
server_key: "f18093d44bac6f0e7a667c5fe28a0b62"
server_key_iv: "ec69810a1b70a8ff09e52d7e2b8de25b"
server_key_version: "encryptionkey@0"
I20260928 13:00:53.498523 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "d7e5820c82904c47b7d3fa91256082c1"
format_stamp: "Formatted at 2026-09-28 13:00:53 on dist-test-slave-8sjz"
server_key: "f18093d44bac6f0e7a667c5fe28a0b62"
server_key_iv: "ec69810a1b70a8ff09e52d7e2b8de25b"
server_key_version: "encryptionkey@0"
I20260928 13:00:53.499856 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:00:53.500697 24942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:53.500927 24927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:53.501037 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "d7e5820c82904c47b7d3fa91256082c1"
format_stamp: "Formatted at 2026-09-28 13:00:53 on dist-test-slave-8sjz"
server_key: "f18093d44bac6f0e7a667c5fe28a0b62"
server_key_iv: "ec69810a1b70a8ff09e52d7e2b8de25b"
server_key_version: "encryptionkey@0"
I20260928 13:00:53.501130 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:53.513062 24927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:53.515666 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:53.515828 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:53.520577 24927 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:38369
I20260928 13:00:53.520609 24994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:38369 every 8 connection(s)
I20260928 13:00:53.520989 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:53.521863 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:53.524344 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1: Bootstrap starting.
I20260928 13:00:53.524981 24995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:53.525051 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 24927
I20260928 13:00:53.525178 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:00:53.525344 24995 log.cc:824] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:53.525413 11309 external_mini_cluster.cc:1428] Setting key dbaab9fe61864524504c5675c8a02148
I20260928 13:00:53.526228 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1: No bootstrap required, opened a new log
I20260928 13:00:53.528970 24995 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } }
I20260928 13:00:53.529150 24995 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:53.529196 24995 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e5820c82904c47b7d3fa91256082c1, State: Initialized, Role: FOLLOWER
I20260928 13:00:53.529342 24995 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } }
I20260928 13:00:53.529448 24995 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:53.529492 24995 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:53.529541 24995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:53.530167 24995 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } }
I20260928 13:00:53.530282 24995 leader_election.cc:302] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7e5820c82904c47b7d3fa91256082c1; no voters:
I20260928 13:00:53.530445 24995 leader_election.cc:288] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:00:53.530534 25000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:53.530679 25000 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [term 1 LEADER]: Becoming Leader. State: Replica: d7e5820c82904c47b7d3fa91256082c1, State: Running, Role: LEADER
I20260928 13:00:53.530759 24995 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:53.530810 25000 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } }
I20260928 13:00:53.531294 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e5820c82904c47b7d3fa91256082c1. Latest consensus state: current_term: 1 leader_uuid: "d7e5820c82904c47b7d3fa91256082c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } } }
I20260928 13:00:53.531307 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7e5820c82904c47b7d3fa91256082c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e5820c82904c47b7d3fa91256082c1" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38369 } } }
I20260928 13:00:53.531457 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:53.531693 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:53.531862 25009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:53.532603 25009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[24914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:53.534204 25009 catalog_manager.cc:1347] Generated new cluster ID: 25aa513966e54b45a8b38f25c9950a60
I20260928 13:00:53.534255 25009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:53.538045 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:53.525964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.11.11.126:38369)
0928 13:00:53.526149 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:53.526159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:53.526196 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:00:53.526736 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:53.526746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:53.526813 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:53.527003 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:53.527586 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:53.529112 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:53.530051 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:53.530728 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:53.531191 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:53.534125 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:53.534130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:53.534130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:53.534155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:53.536200 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:53.536619 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:53.536622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:53.536623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:53.536743 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:53.536950 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:53.536951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:53.536951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:53.537276 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:53.537449 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:53.537537 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 13:00:53.537813 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 13:00:53.539041 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:00:53.543643 25009 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:00:53.544250 25009 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:53.559901 25009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7e5820c82904c47b7d3fa91256082c1: Generated new TSK 0
I20260928 13:00:53.560185 25009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:00:53.593426 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 24927
2026-09-28T13:00:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1371 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[25022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[25022](info): set up 2 sockets
Sep 28 13:00:53 dist-test-slave-8sjz krb5kdc[25022](info): commencing operation
krb5kdc: starting...
W20260928 13:00:54.625367 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:00:54 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:00:54.766734 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:54.849454 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:54.849673 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:54.849697 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:54.850904 25035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:54.850950 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:54.850963 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:54.850977 25035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:54.851037 25035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:54.852629 25035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:54.852902 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:54.853137 25035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:54.857999 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:54.858135 25035 server_base.cc:1034] running on GCE node
W20260928 13:00:54.857978 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:54.857975 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:54.858495 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:54.858784 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:54.859967 25035 hybrid_clock.cc:648] HybridClock initialized: now 1790600454859957 us; error 30 us; skew 500 ppm
Sep 28 13:00:54 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600454, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:54.862516 25035 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:54.863003 25035 webserver.cc:460] Webserver started at http://127.11.11.126:33937/ using document root <none> and password file <none>
I20260928 13:00:54.863250 25035 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:54.863322 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:54.863468 25035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:54.864594 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "be3fb982b9044371a83781a9583bbd12"
format_stamp: "Formatted at 2026-09-28 13:00:54 on dist-test-slave-8sjz"
server_key: "753347420ed88df655e50eb4abab0622"
server_key_iv: "0d45995075140cb64f0893624cca2f66"
server_key_version: "encryptionkey@0"
I20260928 13:00:54.864964 25035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "be3fb982b9044371a83781a9583bbd12"
format_stamp: "Formatted at 2026-09-28 13:00:54 on dist-test-slave-8sjz"
server_key: "753347420ed88df655e50eb4abab0622"
server_key_iv: "0d45995075140cb64f0893624cca2f66"
server_key_version: "encryptionkey@0"
I20260928 13:00:54.866292 25035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:00:54.867005 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:54.867177 25035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:00:54.867285 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "be3fb982b9044371a83781a9583bbd12"
format_stamp: "Formatted at 2026-09-28 13:00:54 on dist-test-slave-8sjz"
server_key: "753347420ed88df655e50eb4abab0622"
server_key_iv: "0d45995075140cb64f0893624cca2f66"
server_key_version: "encryptionkey@0"
I20260928 13:00:54.867360 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:54.881474 25035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:54.883394 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:54.883570 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:54.887909 25035 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41711
I20260928 13:00:54.887945 25102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41711 every 8 connection(s)
I20260928 13:00:54.888365 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:54.889382 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:54.891736 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap starting.
I20260928 13:00:54.892004 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25035
I20260928 13:00:54.892138 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:00:54.892367 11309 external_mini_cluster.cc:1428] Setting key 5f196d6824f2a7dc7fcf249e81812c08
I20260928 13:00:54.892556 25103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:54.892886 25103 log.cc:824] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:54.893723 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: No bootstrap required, opened a new log
I20260928 13:00:54.896076 25103 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:54.896245 25103 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:54.896281 25103 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Initialized, Role: FOLLOWER
I20260928 13:00:54.896431 25103 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:54.896525 25103 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:54.896549 25103 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:54.896605 25103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:54.897193 25103 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:54.897295 25103 leader_election.cc:302] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3fb982b9044371a83781a9583bbd12; no voters:
I20260928 13:00:54.897450 25103 leader_election.cc:288] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:00:54.897511 25108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:54.897680 25108 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 LEADER]: Becoming Leader. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Running, Role: LEADER
I20260928 13:00:54.897689 25103 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:54.897802 25108 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:54.898103 25109 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:54.898186 25109 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:54.898437 25113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:54.898958 25113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:54.899492 25110 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb982b9044371a83781a9583bbd12. Latest consensus state: current_term: 1 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:54.899591 25110 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
Sep 28 13:00:54 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:54.901131 25113 catalog_manager.cc:1347] Generated new cluster ID: b79b502bd0a84152bfa54e903bd369c9
I20260928 13:00:54.901192 25113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:54.904604 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:54.893052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.11.11.126:41711)
0928 13:00:54.893309 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:54.893317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:54.893341 (+ 24us) server_negotiation.cc:407] Connection header received
0928 13:00:54.893978 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:54.893987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:54.894050 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:54.894211 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:54.894819 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:54.896263 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:54.897137 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:54.898036 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:54.898175 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:54.901098 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:54.901102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:54.901102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:54.901115 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:54.902961 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:54.903362 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:54.903365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:54.903366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:54.903464 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:54.903678 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:54.903679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:54.903679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:54.903935 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:54.904153 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:54.904247 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 13:00:54.904392 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":181,"threads_started":1}
I20260928 13:00:54.916513 25113 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:00:54.917088 25113 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:54.923731 25113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Generated new TSK 0
I20260928 13:00:54.923949 25113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:00:54.955207 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:55.045425 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.045640 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.045665 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.046833 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.046885 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.046912 25129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:00:55.048473 25129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.048731 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.048979 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.054014 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.054000 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.054105 25129 server_base.cc:1034] running on GCE node
W20260928 13:00:55.054072 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.054438 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.054672 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.055811 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790600455055791 us; error 37 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.058378 25129 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:00:55.058838 25129 webserver.cc:460] Webserver started at http://127.11.11.65:34457/ using document root <none> and password file <none>
I20260928 13:00:55.059069 25129 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.059113 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.059257 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:55.060451 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "2ba9579526804314a564832703be757b"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "59a9e36e6f2e65760e27233196d931c2"
server_key_iv: "208c784ce1dbed722f84756bfee5b9ca"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.060808 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "2ba9579526804314a564832703be757b"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "59a9e36e6f2e65760e27233196d931c2"
server_key_iv: "208c784ce1dbed722f84756bfee5b9ca"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.062121 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:00:55.062857 25144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.063027 25129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:00:55.063144 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "2ba9579526804314a564832703be757b"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "59a9e36e6f2e65760e27233196d931c2"
server_key_iv: "208c784ce1dbed722f84756bfee5b9ca"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.063218 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.075232 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.077732 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.077905 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.078150 25129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:55.078473 25129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:55.078532 25129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.078576 25129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:55.078604 25129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.084514 25129 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:34397
I20260928 13:00:55.084556 25257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:34397 every 8 connection(s)
I20260928 13:00:55.084914 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:00:55.090813 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25129
I20260928 13:00:55.090939 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:55.091135 11309 external_mini_cluster.cc:1428] Setting key 7383c94445044f5c240d091bbcf31be8
I20260928 13:00:55.094949 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.085703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:36293 (local address 127.11.11.126:41711)
0928 13:00:55.085766 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.085768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.085962 (+ 194us) server_negotiation.cc:407] Connection header received
0928 13:00:55.086467 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.086468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.086477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.086535 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.087481 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.087988 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.089071 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.089288 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.089349 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.091716 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.091718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.091719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.091727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.093466 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.093914 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.093916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.093917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.093981 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.094246 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.094248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.094248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.094377 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.094457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.094788 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.094876 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:00:55.095240 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.085673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:36293)
0928 13:00:55.085830 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.085874 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.086378 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.086599 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.086601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.087184 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.087391 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.087396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.088170 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.088171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.088989 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.088991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.089380 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.089381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.089481 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.090074 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.090082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.091609 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.093556 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.093565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.093567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.093828 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.094058 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.094059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.094059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.094176 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.094449 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.094455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.094633 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.094749 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.094950 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 13:00:55.095813 25258 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:55.095950 25258 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:55.096186 25258 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:55.096799 25067 ts_manager.cc:194] Registered new tserver with Master: 2ba9579526804314a564832703be757b (127.11.11.65:34397)
I20260928 13:00:55.097494 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:36293
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:00:55.119439 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:55.202288 25263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.202553 25263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.202593 25263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.203847 25263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.203914 25263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.203961 25263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:00:55.205530 25263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.205808 25263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.206056 25263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.211185 25269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.211138 25271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.211375 25263 server_base.cc:1034] running on GCE node
W20260928 13:00:55.211138 25268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.211663 25263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.211910 25263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.213078 25263 hybrid_clock.cc:648] HybridClock initialized: now 1790600455213073 us; error 36 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.215754 25263 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:00:55.216303 25263 webserver.cc:460] Webserver started at http://127.11.11.66:43821/ using document root <none> and password file <none>
I20260928 13:00:55.216542 25263 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.216612 25263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.216753 25263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:55.217892 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "a5477bd9abe3404a9e12f5fa45179735"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "422ff49695a9fcd7432f1fe28af713f7"
server_key_iv: "d9a9662dd1bd935813c7a26818d7fb70"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.218286 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "a5477bd9abe3404a9e12f5fa45179735"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "422ff49695a9fcd7432f1fe28af713f7"
server_key_iv: "d9a9662dd1bd935813c7a26818d7fb70"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.219621 25263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:00:55.220438 25278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.220613 25263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.220717 25263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "a5477bd9abe3404a9e12f5fa45179735"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "422ff49695a9fcd7432f1fe28af713f7"
server_key_iv: "d9a9662dd1bd935813c7a26818d7fb70"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.220795 25263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.233520 25263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.235491 25263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.235664 25263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.235909 25263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:55.236295 25263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:55.236349 25263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.236392 25263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:55.236419 25263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.242543 25263 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:32889
I20260928 13:00:55.242589 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:32889 every 8 connection(s)
I20260928 13:00:55.242938 25263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:00:55.244179 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25263
I20260928 13:00:55.244256 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:00:55.244458 11309 external_mini_cluster.cc:1428] Setting key 6805debcbf83d6fd690535c8a0dd39dd
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 13:00:55.252822 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.243701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:53745 (local address 127.11.11.126:41711)
0928 13:00:55.243746 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.243748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.243993 (+ 245us) server_negotiation.cc:407] Connection header received
0928 13:00:55.244616 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.244618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.244632 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.244691 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.245671 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.246175 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.247185 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.247408 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.247486 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.249859 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.249860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.249861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.249869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.251363 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.251779 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.251780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.251781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.251839 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.252173 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.252174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.252174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.252295 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.252376 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.252646 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.252732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:00:55.253021 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.243677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:53745)
0928 13:00:55.243830 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.243867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.244513 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.244787 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.244796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.245367 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.245563 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.245567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.246274 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.246275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.247085 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.247087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.247492 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.247494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.247622 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.248374 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.248385 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.249760 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.251449 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.251456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.251458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.251686 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.251905 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.251906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.251906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.252017 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.252366 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.252373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.252514 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.252613 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.252783 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260928 13:00:55.253523 25392 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:55.253635 25392 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:55.253813 25392 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:55.254299 25067 ts_manager.cc:194] Registered new tserver with Master: a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
I20260928 13:00:55.254649 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:53745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:00:55.271585 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:55.353616 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.353832 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.353857 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.354961 25397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.355002 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.355029 25397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:00:55.356475 25397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.356671 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.356858 25397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.362036 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.362041 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.362035 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.362361 25397 server_base.cc:1034] running on GCE node
I20260928 13:00:55.362579 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.362843 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.363994 25397 hybrid_clock.cc:648] HybridClock initialized: now 1790600455363973 us; error 38 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.366896 25397 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:00:55.367437 25397 webserver.cc:460] Webserver started at http://127.11.11.67:33289/ using document root <none> and password file <none>
I20260928 13:00:55.367690 25397 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.367760 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.367914 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:00:55.369065 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "189667c4b33641578049ab49a4807f1d"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "ffa8793103ea907ab09f4190a073c17b"
server_key_iv: "24458ca1875d7ae4aba8a934671a84a9"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.369452 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "189667c4b33641578049ab49a4807f1d"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "ffa8793103ea907ab09f4190a073c17b"
server_key_iv: "24458ca1875d7ae4aba8a934671a84a9"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.370919 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:00:55.371701 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.371901 25397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:55.371965 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "189667c4b33641578049ab49a4807f1d"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "ffa8793103ea907ab09f4190a073c17b"
server_key_iv: "24458ca1875d7ae4aba8a934671a84a9"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.372089 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.394327 25397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.396808 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.396971 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.397236 25397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:55.397609 25397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:55.397646 25397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.397701 25397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:55.397728 25397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.404022 25397 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35305
I20260928 13:00:55.404079 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35305 every 8 connection(s)
I20260928 13:00:55.404443 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:00:55.406909 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25397
I20260928 13:00:55.407006 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:00:55.407186 11309 external_mini_cluster.cc:1428] Setting key d582531b29c0ba509ab56bba8a59eb51
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:55.414647 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.405233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:38773 (local address 127.11.11.126:41711)
0928 13:00:55.405314 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.405317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.405531 (+ 214us) server_negotiation.cc:407] Connection header received
0928 13:00:55.406000 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.406001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.406011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.406065 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.407355 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.408224 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.409315 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.409513 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.409573 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.411820 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.411821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.411821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.411828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.413288 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.413748 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.413749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.413750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.413847 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.414085 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.414086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.414086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.414216 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.414273 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.414514 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.414577 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:00:55.414856 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.405204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:38773)
0928 13:00:55.405398 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.405438 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.405920 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.406249 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.406253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.406916 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.407232 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.407239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.408318 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.408319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.409221 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.409224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.409608 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.409608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.409723 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.410305 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.410313 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.411732 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.413379 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.413385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.413386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.413608 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.413919 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.413920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.413920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.414023 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.414275 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.414279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.414403 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.414486 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.414608 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260928 13:00:55.415432 25526 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:55.415548 25526 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:55.415740 25526 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:55.416157 25067 ts_manager.cc:194] Registered new tserver with Master: 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305)
I20260928 13:00:55.416572 25067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:38773
I20260928 13:00:55.418884 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:00:55.426607 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.420262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.11.11.126:41711)
0928 13:00:55.420327 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.420329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.420428 (+ 99us) server_negotiation.cc:407] Connection header received
0928 13:00:55.420468 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.420469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.420477 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.420530 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.421046 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.421524 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.422246 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.422428 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.422477 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.423809 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.423810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.423811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.423818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.425427 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.425854 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.425854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.425855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.425906 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.426096 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.426096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.426096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.426241 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.426319 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.426476 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.426540 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:00:55.428488 25067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:00:55.430179 25067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:00:55.437479 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.432728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.11.11.65:34397)
0928 13:00:55.432933 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.432938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.433041 (+ 103us) server_negotiation.cc:407] Connection header received
0928 13:00:55.433191 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.433194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.433216 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.433299 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:55.433482 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.434659 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.436320 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.437086 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.437165 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.437226 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.437355 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.437398 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260928 13:00:55.437481 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.432624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:34397 (local address 127.0.0.1:56674)
0928 13:00:55.432922 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.432969 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.433076 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.433317 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.433320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.433322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:55.433397 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.433399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.434753 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.434753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.436233 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.436236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.437168 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.437169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.437247 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.437264 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.437339 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.437398 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":45,"thread_start_us":67,"threads_started":1}
I20260928 13:00:55.438409 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.432867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32889 (local address 127.0.0.1:46996)
0928 13:00:55.432993 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.433001 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.434772 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.434966 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.434967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.434969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:55.435119 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.435120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.436031 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.436032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.437186 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.437189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.438121 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.438122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.438217 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.438221 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.438293 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.438335 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":89,"thread_start_us":78,"threads_started":1}
I20260928 13:00:55.438452 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.433090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.11.11.66:32889)
0928 13:00:55.433250 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.433263 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.433275 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:00:55.434855 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.434858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.434881 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.434970 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:55.435193 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.435928 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.437265 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.438035 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.438118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.438173 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.438311 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.438370 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260928 13:00:55.438611 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.432657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35305 (local address 127.0.0.1:58998)
0928 13:00:55.432989 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.433000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.433070 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.433442 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.433443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.433445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:55.433485 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.433486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.434751 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.434752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.436233 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.436236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.438225 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.438226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.438300 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.438304 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.438358 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.438401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":101,"thread_start_us":86,"threads_started":1}
I20260928 13:00:55.438695 25192 tablet_service.cc:1467] Processing CreateTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 (DEFAULT_TABLE table=test-table [id=1fe821b6120f406db5956a839e231cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:55.438997 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.432830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.11.11.67:35305)
0928 13:00:55.433306 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.433310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.433319 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:00:55.433362 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.433364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.433376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.433465 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:55.433543 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.434635 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.437266 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.437946 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.438024 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.438079 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.438457 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.438932 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":79,"threads_started":1}
I20260928 13:00:55.439018 25192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94fb8ac924c49c3b4234b390bc6b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:55.440732 25326 tablet_service.cc:1467] Processing CreateTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 (DEFAULT_TABLE table=test-table [id=1fe821b6120f406db5956a839e231cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:55.440785 25460 tablet_service.cc:1467] Processing CreateTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 (DEFAULT_TABLE table=test-table [id=1fe821b6120f406db5956a839e231cc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:55.441079 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94fb8ac924c49c3b4234b390bc6b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:55.441119 25460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94fb8ac924c49c3b4234b390bc6b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:55.442397 25543 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Bootstrap starting.
I20260928 13:00:55.443231 25543 tablet_bootstrap.cc:654] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:55.443944 25543 log.cc:824] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:55.444875 25543 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: No bootstrap required, opened a new log
I20260928 13:00:55.444960 25543 ts_tablet_manager.cc:1397] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:00:55.445024 25545 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Bootstrap starting.
I20260928 13:00:55.445719 25545 tablet_bootstrap.cc:654] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:55.446339 25545 log.cc:824] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:55.447177 25545 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: No bootstrap required, opened a new log
I20260928 13:00:55.447266 25545 ts_tablet_manager.cc:1397] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:00:55.447247 25543 raft_consensus.cc:348] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.447443 25543 raft_consensus.cc:374] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:55.447481 25543 raft_consensus.cc:729] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba9579526804314a564832703be757b, State: Initialized, Role: FOLLOWER
I20260928 13:00:55.447608 25543 consensus_queue.cc:260] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.447903 25543 ts_tablet_manager.cc:1428] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:00:55.447911 25258 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
I20260928 13:00:55.448182 25546 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Bootstrap starting.
I20260928 13:00:55.448967 25546 tablet_bootstrap.cc:654] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:55.449577 25546 log.cc:824] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:55.449642 25545 raft_consensus.cc:348] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.449832 25545 raft_consensus.cc:374] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:55.449893 25545 raft_consensus.cc:729] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189667c4b33641578049ab49a4807f1d, State: Initialized, Role: FOLLOWER
I20260928 13:00:55.450022 25545 consensus_queue.cc:260] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.450332 25545 ts_tablet_manager.cc:1428] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:00:55.450371 25526 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
I20260928 13:00:55.450474 25546 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: No bootstrap required, opened a new log
I20260928 13:00:55.450531 25546 ts_tablet_manager.cc:1397] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:00:55.452535 25546 raft_consensus.cc:348] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.452759 25546 raft_consensus.cc:374] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:55.452806 25546 raft_consensus.cc:729] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5477bd9abe3404a9e12f5fa45179735, State: Initialized, Role: FOLLOWER
I20260928 13:00:55.452942 25546 consensus_queue.cc:260] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.453258 25546 ts_tablet_manager.cc:1428] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:00:55.453408 25392 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
W20260928 13:00:55.493589 25393 tablet.cc:2369] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:00:55.497368 25551 raft_consensus.cc:482] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:00:55.497493 25551 raft_consensus.cc:504] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.497912 25551 leader_election.cc:288] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ba9579526804314a564832703be757b (127.11.11.65:34397), 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305)
I20260928 13:00:55.501219 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.498312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49427 (local address 127.11.11.67:35305)
0928 13:00:55.498380 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.498382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.498391 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:00:55.498426 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.498427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.498434 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.498508 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:55.498705 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.499441 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500444 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.500846 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500937 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.500947 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.501105 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.501144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:00:55.501219 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.498087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35305 (local address 127.11.11.66:49427)
0928 13:00:55.498150 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.498165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.498238 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.498516 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.498518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.498520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:55.498594 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.498596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.499520 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.499521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.500353 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.500356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.500933 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.501021 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.501030 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.501099 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.501144 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 13:00:55.501303 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.498181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:57771 (local address 127.11.11.65:34397)
0928 13:00:55.498228 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.498230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.498395 (+ 165us) server_negotiation.cc:407] Connection header received
0928 13:00:55.498453 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.498454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.498465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.498545 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:55.498733 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.499308 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500465 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.500891 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500994 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.501003 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.501168 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.501221 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:00:55.501359 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.498087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:34397 (local address 127.11.11.66:57771)
0928 13:00:55.498296 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.498306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.498395 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.498548 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.498549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.498551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:55.498634 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.498636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.499400 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.499401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.500358 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.500361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.500994 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.500995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.501081 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.501085 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.501160 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.501204 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":62,"thread_start_us":113,"threads_started":1}
I20260928 13:00:55.501677 25212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94fb8ac924c49c3b4234b390bc6b2f4" candidate_uuid: "a5477bd9abe3404a9e12f5fa45179735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba9579526804314a564832703be757b" is_pre_election: true
I20260928 13:00:55.501791 25480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94fb8ac924c49c3b4234b390bc6b2f4" candidate_uuid: "a5477bd9abe3404a9e12f5fa45179735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189667c4b33641578049ab49a4807f1d" is_pre_election: true
I20260928 13:00:55.501878 25212 raft_consensus.cc:2454] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5477bd9abe3404a9e12f5fa45179735 in term 0.
I20260928 13:00:55.501986 25480 raft_consensus.cc:2454] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5477bd9abe3404a9e12f5fa45179735 in term 0.
I20260928 13:00:55.502125 25280 leader_election.cc:302] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba9579526804314a564832703be757b, a5477bd9abe3404a9e12f5fa45179735; no voters:
I20260928 13:00:55.502259 25551 raft_consensus.cc:2781] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:00:55.502308 25551 raft_consensus.cc:482] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:00:55.502324 25551 raft_consensus.cc:3037] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:55.503028 25551 raft_consensus.cc:504] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.503198 25551 leader_election.cc:288] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [CANDIDATE]: Term 1 election: Requested vote from peers 2ba9579526804314a564832703be757b (127.11.11.65:34397), 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305)
I20260928 13:00:55.503340 25212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94fb8ac924c49c3b4234b390bc6b2f4" candidate_uuid: "a5477bd9abe3404a9e12f5fa45179735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba9579526804314a564832703be757b"
I20260928 13:00:55.503398 25480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94fb8ac924c49c3b4234b390bc6b2f4" candidate_uuid: "a5477bd9abe3404a9e12f5fa45179735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189667c4b33641578049ab49a4807f1d"
I20260928 13:00:55.503445 25212 raft_consensus.cc:3037] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:55.503495 25480 raft_consensus.cc:3037] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:55.504285 25212 raft_consensus.cc:2454] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5477bd9abe3404a9e12f5fa45179735 in term 1.
I20260928 13:00:55.504432 25480 raft_consensus.cc:2454] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5477bd9abe3404a9e12f5fa45179735 in term 1.
I20260928 13:00:55.504449 25280 leader_election.cc:302] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba9579526804314a564832703be757b, a5477bd9abe3404a9e12f5fa45179735; no voters:
I20260928 13:00:55.504558 25551 raft_consensus.cc:2781] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:55.504745 25551 raft_consensus.cc:686] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 1 LEADER]: Becoming Leader. State: Replica: a5477bd9abe3404a9e12f5fa45179735, State: Running, Role: LEADER
I20260928 13:00:55.504854 25551 consensus_queue.cc:237] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.505698 25067 catalog_manager.cc:5495] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66). New cstate: current_term: 1 leader_uuid: "a5477bd9abe3404a9e12f5fa45179735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } health_report { overall_health: UNKNOWN } } }
I20260928 13:00:55.516160 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.513448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.11.11.66:32889)
0928 13:00:55.513516 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.513518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.513610 (+ 92us) server_negotiation.cc:407] Connection header received
0928 13:00:55.513649 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.513650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.513667 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.513742 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:00:55.513903 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.514426 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.515277 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.515499 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.515588 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.515767 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:00:55.515877 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:00:55.515965 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.516004 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.516025 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:00:55.518065 25480 raft_consensus.cc:1261] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Refusing update from remote peer a5477bd9abe3404a9e12f5fa45179735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:55.518054 25212 raft_consensus.cc:1261] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Refusing update from remote peer a5477bd9abe3404a9e12f5fa45179735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:55.518435 25551 consensus_queue.cc:1035] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:55.518579 25553 consensus_queue.cc:1035] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:55.521577 25558 mvcc.cc:205] Tried to move back new op lower bound from 7334299465799245824 to 7334299465748566016. Current Snapshot: MvccSnapshot[applied={T|T < 7334299465799245824 or (T in {7334299465799245824})}]
I20260928 13:00:55.521895 25557 mvcc.cc:205] Tried to move back new op lower bound from 7334299465799245824 to 7334299465748566016. Current Snapshot: MvccSnapshot[applied={T|T < 7334299465799245824}]
I20260928 13:00:55.523895 25067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:55.524000 25067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:55.524816 25067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Sending DeleteTablet for 3 replicas of tablet f94fb8ac924c49c3b4234b390bc6b2f4
I20260928 13:00:55.525601 25460 tablet_service.cc:1514] Processing DeleteTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:55 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:58998
I20260928 13:00:55.525825 25565 tablet_replica.cc:330] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: stopping tablet replica
I20260928 13:00:55.525774 25326 tablet_service.cc:1514] Processing DeleteTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:55 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:46996
I20260928 13:00:55.525918 25565 raft_consensus.cc:2229] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:55.526012 25565 raft_consensus.cc:2258] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:55.526324 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25129
I20260928 13:00:55.526398 25192 tablet_service.cc:1514] Processing DeleteTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:55 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:56674
I20260928 13:00:55.527805 25565 ts_tablet_manager.cc:1905] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:55.529603 25565 ts_tablet_manager.cc:1918] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:55.529716 25565 log.cc:1196] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/f94fb8ac924c49c3b4234b390bc6b2f4
I20260928 13:00:55.529982 25565 ts_tablet_manager.cc:1939] T f94fb8ac924c49c3b4234b390bc6b2f4 P 189667c4b33641578049ab49a4807f1d: Deleting consensus metadata
I20260928 13:00:55.530402 25054 catalog_manager.cc:4841] TS 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): tablet f94fb8ac924c49c3b4234b390bc6b2f4 (table test-table [id=1fe821b6120f406db5956a839e231cc4]) successfully deleted
I20260928 13:00:55.532198 25566 tablet_replica.cc:330] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: stopping tablet replica
I20260928 13:00:55.532342 25566 raft_consensus.cc:2229] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:00:55.532480 25566 raft_consensus.cc:2258] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:55.532904 25566 ts_tablet_manager.cc:1905] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 13:00:55.533402 25052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:34397 (error 108)
I20260928 13:00:55.533535 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25263
I20260928 13:00:55.534457 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.534321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:34397 (local address 127.0.0.1:56688)
0928 13:00:55.534379 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.534418 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:34397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 13:00:55.534612 25052 catalog_manager.cc:4584] TS 2ba9579526804314a564832703be757b (127.11.11.65:34397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f94fb8ac924c49c3b4234b390bc6b2f4: Network error: Client connection negotiation failed: client connection to 127.11.11.65:34397: connect: Connection refused (error 111)
I20260928 13:00:55.540799 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25397
I20260928 13:00:55.541013 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.540838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32889 (local address 127.0.0.1:47016)
0928 13:00:55.540940 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.540971 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.66:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 13:00:55.545825 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25035
I20260928 13:00:55.551395 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--webserver_interface=127.11.11.126
--webserver_port=33937
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:55.635195 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.635432 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.635455 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.636756 25568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:55.636809 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.636824 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.636837 25568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:55.636850 25568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:55.638444 25568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.638739 25568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.638998 25568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.647734 25577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.647780 25575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.647783 25574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.647835 25568 server_base.cc:1034] running on GCE node
I20260928 13:00:55.648268 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.648612 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.649806 25568 hybrid_clock.cc:648] HybridClock initialized: now 1790600455649796 us; error 34 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.652415 25568 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:55.652930 25568 webserver.cc:460] Webserver started at http://127.11.11.126:33937/ using document root <none> and password file <none>
I20260928 13:00:55.653187 25568 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.653281 25568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.654686 25568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:55.655421 25584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.655646 25568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:55.655761 25568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "be3fb982b9044371a83781a9583bbd12"
format_stamp: "Formatted at 2026-09-28 13:00:54 on dist-test-slave-8sjz"
server_key: "753347420ed88df655e50eb4abab0622"
server_key_iv: "0d45995075140cb64f0893624cca2f66"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.656261 25568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.671525 25568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.673918 25568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.674108 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.678813 25568 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41711
I20260928 13:00:55.678848 25636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41711 every 8 connection(s)
I20260928 13:00:55.679304 25568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:55.681039 25637 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:00:55.681648 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap starting.
I20260928 13:00:55.686831 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25568
I20260928 13:00:55.687129 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:34397
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=34457
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:55.690096 25637 log.cc:824] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:55.692189 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:00:55.692510 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap complete.
I20260928 13:00:55.694826 25637 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:55.695082 25637 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Initialized, Role: FOLLOWER
I20260928 13:00:55.695279 25637 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:55.695366 25637 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:55.695425 25637 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:55.695477 25637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:00:55.696611 25637 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:55.696733 25637 leader_election.cc:302] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3fb982b9044371a83781a9583bbd12; no voters:
I20260928 13:00:55.696943 25637 leader_election.cc:288] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:00:55.696979 25641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:00:55.697091 25641 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 LEADER]: Becoming Leader. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Running, Role: LEADER
I20260928 13:00:55.697184 25641 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:55.697223 25637 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:55.697431 25641 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb982b9044371a83781a9583bbd12. Latest consensus state: current_term: 2 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:55.697507 25641 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:55.697434 25642 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:55.697690 25642 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:55.697690 25645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:55.698597 25645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:55.698911 25645 catalog_manager.cc:1259] Loaded cluster ID: b79b502bd0a84152bfa54e903bd369c9
I20260928 13:00:55.699080 25645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:00:55.699342 25645 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 13:00:55.704481 25645 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 13:00:55.707806 25645 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:00:55.708026 25645 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Loaded TSK: 0
I20260928 13:00:55.708292 25645 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:00:55.774573 25639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.774794 25639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.774820 25639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.776029 25639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.776132 25639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.776161 25639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:00:55.777650 25639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:34397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=34457
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.777912 25639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.778167 25639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.784171 25664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.784127 25665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.784124 25667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.784566 25639 server_base.cc:1034] running on GCE node
I20260928 13:00:55.784775 25639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.785015 25639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.786172 25639 hybrid_clock.cc:648] HybridClock initialized: now 1790600455786159 us; error 30 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.788727 25639 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:00:55.789222 25639 webserver.cc:460] Webserver started at http://127.11.11.65:34457/ using document root <none> and password file <none>
I20260928 13:00:55.789472 25639 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.789530 25639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.790921 25639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:00:55.791631 25674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.791777 25639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:00:55.791865 25639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "2ba9579526804314a564832703be757b"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "59a9e36e6f2e65760e27233196d931c2"
server_key_iv: "208c784ce1dbed722f84756bfee5b9ca"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.792315 25639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.804504 25639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.806905 25639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.807055 25639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.807335 25639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:55.807859 25681 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:00:55.809329 25639 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:00:55.809381 25639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:00:55.809418 25639 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:00:55.810092 25639 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:00:55.810137 25639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:00:55.810184 25681 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Bootstrap starting.
I20260928 13:00:55.818058 25639 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:34397
I20260928 13:00:55.818228 25788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:34397 every 8 connection(s)
I20260928 13:00:55.818640 25639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:00:55.822264 25681 log.cc:824] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:55.822600 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25639
I20260928 13:00:55.822903 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:32889
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=43821
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:55.824793 25681 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:00:55.825098 25681 tablet_bootstrap.cc:492] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Bootstrap complete.
I20260928 13:00:55.826090 25681 ts_tablet_manager.cc:1397] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.001s
I20260928 13:00:55.827323 25681 raft_consensus.cc:348] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.827654 25681 raft_consensus.cc:729] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba9579526804314a564832703be757b, State: Initialized, Role: FOLLOWER
I20260928 13:00:55.827832 25681 consensus_queue.cc:260] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } }
I20260928 13:00:55.828169 25681 ts_tablet_manager.cc:1428] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:55.834684 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.819572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:56595 (local address 127.11.11.126:41711)
0928 13:00:55.819823 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.819860 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.820021 (+ 161us) server_negotiation.cc:407] Connection header received
0928 13:00:55.820986 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.820994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.821057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.821208 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.822210 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.825328 (+ 3118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.826876 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.827489 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.827610 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.830496 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.830501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.830501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.830515 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.832686 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.833305 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.833308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.833309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.833403 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.833727 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.833727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.833728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.833956 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.834020 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.834415 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.834528 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260928 13:00:55.834858 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.819447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:56595)
0928 13:00:55.819825 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.819885 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.820718 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.821298 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.821304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.821766 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.822054 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.822064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.825478 (+ 3414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.825480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.826771 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.826774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.827589 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.827590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.827723 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.828389 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.828399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.830374 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.832806 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.832816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.832818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.833190 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.833476 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.833477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.833477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.833646 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.834029 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.834037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.834233 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.834367 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.834586 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":90,"threads_started":1}
I20260928 13:00:55.835582 25789 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:55.835696 25789 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:55.835891 25789 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:55.836675 25601 ts_manager.cc:194] Registered new tserver with Master: 2ba9579526804314a564832703be757b (127.11.11.65:34397)
I20260928 13:00:55.837719 25601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:56595
I20260928 13:00:55.838784 25789 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.65@KRBTEST.COM
I20260928 13:00:55.846832 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.837319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:34397 (local address 127.0.0.1:56694)
0928 13:00:55.838062 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.838097 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.838161 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.838462 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.838468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.838805 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.838913 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.838916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.840240 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.840240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.840871 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.840873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.841499 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.841500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.841560 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.842082 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.842086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.843575 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.845766 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.845773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.845775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.845971 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.846185 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.846186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.846186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.846239 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.846533 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.846538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.846582 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.846646 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.846748 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":91,"threads_started":1}
I20260928 13:00:55.846899 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.837392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.11.11.65:34397)
0928 13:00:55.838306 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.838310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.838319 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:00:55.838352 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.838354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.838368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.838438 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.839006 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.840151 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.840953 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.841420 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.841491 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.843660 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.843668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.843669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.843676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.845660 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.846049 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.846052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.846053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.846120 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.846300 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.846301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.846301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.846469 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.846551 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.846700 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.846796 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":94,"threads_started":1}
I20260928 13:00:55.847324 25723 tablet_service.cc:1514] Processing DeleteTablet for tablet f94fb8ac924c49c3b4234b390bc6b2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:55 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:56694
I20260928 13:00:55.847862 25798 tablet_replica.cc:330] stopping tablet replica
I20260928 13:00:55.847971 25798 raft_consensus.cc:2229] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:55.848066 25798 raft_consensus.cc:2258] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:55.848428 25798 ts_tablet_manager.cc:1905] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:55.850519 25798 ts_tablet_manager.cc:1918] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:55.850589 25798 log.cc:1196] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/f94fb8ac924c49c3b4234b390bc6b2f4
I20260928 13:00:55.850828 25798 ts_tablet_manager.cc:1939] T f94fb8ac924c49c3b4234b390bc6b2f4 P 2ba9579526804314a564832703be757b: Deleting consensus metadata
I20260928 13:00:55.851380 25586 catalog_manager.cc:4841] TS 2ba9579526804314a564832703be757b (127.11.11.65:34397): tablet f94fb8ac924c49c3b4234b390bc6b2f4 (table test-table [id=1fe821b6120f406db5956a839e231cc4]) successfully deleted
W20260928 13:00:55.914913 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:55.915164 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:55.915206 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:55.916419 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:55.916491 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:55.916538 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:00:55.918053 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=43821
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:55.918341 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:55.918586 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:55.923789 25803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.923799 25806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:55.923774 25804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:55.924191 25794 server_base.cc:1034] running on GCE node
I20260928 13:00:55.924444 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:55.924717 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:55.925879 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790600455925869 us; error 31 us; skew 500 ppm
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:55.928517 25794 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:00:55.928983 25794 webserver.cc:460] Webserver started at http://127.11.11.66:43821/ using document root <none> and password file <none>
I20260928 13:00:55.929236 25794 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:55.929301 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:55.930699 25794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:55.931458 25813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.931682 25794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:55.931751 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "a5477bd9abe3404a9e12f5fa45179735"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "422ff49695a9fcd7432f1fe28af713f7"
server_key_iv: "d9a9662dd1bd935813c7a26818d7fb70"
server_key_version: "encryptionkey@0"
I20260928 13:00:55.932209 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:55.949445 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:55.951521 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:55.951690 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:55.951941 25794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:55.952430 25820 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 13:00:55.953879 25820 ts_tablet_manager.cc:1852] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 13:00:55.954159 25820 ts_tablet_manager.cc:1905] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:55.956283 25820 ts_tablet_manager.cc:1918] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:55.956367 25820 log.cc:1196] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/f94fb8ac924c49c3b4234b390bc6b2f4
I20260928 13:00:55.956584 25820 ts_tablet_manager.cc:1939] T f94fb8ac924c49c3b4234b390bc6b2f4 P a5477bd9abe3404a9e12f5fa45179735: Deleting consensus metadata
I20260928 13:00:55.956764 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 13:00:55.956831 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 13:00:55.956882 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:55.956914 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:55.963464 25794 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:32889
I20260928 13:00:55.963518 25927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:32889 every 8 connection(s)
I20260928 13:00:55.963860 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:00:55 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600455, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:55.973093 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.964661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:49195 (local address 127.11.11.126:41711)
0928 13:00:55.964712 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:55.964714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:55.964944 (+ 230us) server_negotiation.cc:407] Connection header received
0928 13:00:55.965532 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:55.965534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:55.965552 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:55.965598 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:55.966256 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.966761 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.967777 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.967990 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.968095 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.970390 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:55.970392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:55.970392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:55.970408 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:55.971607 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.972020 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.972021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.972022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.972128 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:55.972399 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:55.972400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:55.972401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:55.972523 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:55.972596 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:55.972898 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 13:00:55.972995 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:00:55.973207 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25794
I20260928 13:00:55.973302 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:55.964637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:49195)
0928 13:00:55.964819 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:55.964855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:55.965412 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:55.965691 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:55.965695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:55.965959 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:55.966157 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.966161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.966867 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.966868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.967683 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:55.967685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:55.968102 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:55.968103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:55.968230 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:55.968749 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:55.968759 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:55.970289 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:55.971701 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.971707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.971710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.971942 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.972203 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:55.972204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:55.972204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:55.972326 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:55.972596 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:55.972604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:55.972746 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 13:00:55.972867 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 13:00:55.973056 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260928 13:00:55.973495 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:35305
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=33289
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:55.974031 25928 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:55.974128 25928 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:55.974349 25928 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:55.974797 25601 ts_manager.cc:194] Registered new tserver with Master: a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
I20260928 13:00:55.975347 25601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:49195
W20260928 13:00:56.055498 25931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:56.055744 25931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:56.055786 25931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:56.057019 25931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:56.057090 25931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:56.057137 25931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:00:56.058678 25931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:35305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=33289
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:56.058959 25931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:56.059216 25931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:56.064369 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.064426 25931 server_base.cc:1034] running on GCE node
W20260928 13:00:56.064399 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.064409 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.064832 25931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:56.065080 25931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:56.066252 25931 hybrid_clock.cc:648] HybridClock initialized: now 1790600456066247 us; error 37 us; skew 500 ppm
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:56.068849 25931 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:00:56.069300 25931 webserver.cc:460] Webserver started at http://127.11.11.67:33289/ using document root <none> and password file <none>
I20260928 13:00:56.069528 25931 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:56.069610 25931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:56.070914 25931 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:56.071507 25946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.071676 25931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:00:56.071779 25931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "189667c4b33641578049ab49a4807f1d"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "ffa8793103ea907ab09f4190a073c17b"
server_key_iv: "24458ca1875d7ae4aba8a934671a84a9"
server_key_version: "encryptionkey@0"
I20260928 13:00:56.072170 25931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:56.085124 25931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:56.087551 25931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:56.087726 25931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:56.087970 25931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:56.088305 25931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:56.088361 25931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.088404 25931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:56.088431 25931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.094798 25931 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35305
I20260928 13:00:56.094992 26059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35305 every 8 connection(s)
I20260928 13:00:56.095345 25931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:00:56.099121 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 25931
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:00:56.106693 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.096293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51583 (local address 127.11.11.126:41711)
0928 13:00:56.096352 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.096354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.096605 (+ 251us) server_negotiation.cc:407] Connection header received
0928 13:00:56.097191 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.097192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.097206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.097254 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:56.098362 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.098858 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.100350 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.100572 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.100661 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.103381 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:56.103382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:56.103383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:56.103391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:56.105136 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:56.105560 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:56.105561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:56.105562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:56.105622 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:56.105898 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:56.105899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:56.105900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:56.106042 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:56.106109 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.106472 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.106574 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:00:56.106914 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.096310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:51583)
0928 13:00:56.096464 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.096504 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.097002 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.097347 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:56.097351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:56.097956 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:56.098160 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.098165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.098965 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.098966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.100252 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.100255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.100672 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.100673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.100804 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.101493 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:56.101504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:56.103251 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:56.105223 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:56.105230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:56.105232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:56.105480 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:56.105719 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:56.105720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:56.105720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:56.105832 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:56.106112 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:56.106118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:56.106310 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 13:00:56.106447 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 13:00:56.106631 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260928 13:00:56.107122 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.099541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.11.11.126:41711)
0928 13:00:56.099816 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.099818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.099825 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:00:56.099858 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.099859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.099870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.099934 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:56.100615 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.101255 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.102404 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.102605 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.102678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.104249 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:56.104250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:56.104250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:56.104257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:56.105702 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:56.106099 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:56.106101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:56.106101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:56.106169 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:56.106369 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:56.106370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:56.106370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:56.106504 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:56.106670 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.106721 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.106931 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":55,"threads_started":1}
I20260928 13:00:56.107399 26060 heartbeater.cc:347] Connected to a master server at 127.11.11.126:41711
I20260928 13:00:56.107527 26060 heartbeater.cc:464] Registering TS with master...
I20260928 13:00:56.107746 26060 heartbeater.cc:511] Master 127.11.11.126:41711 requested a full tablet report, sending...
I20260928 13:00:56.108196 25601 ts_manager.cc:194] Registered new tserver with Master: 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305)
I20260928 13:00:56.108528 25601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:51583
I20260928 13:00:56.119045 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:00:56.122360 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.119463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.11.11.126:41711)
0928 13:00:56.119586 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.119588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.119600 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:00:56.119675 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.119676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.119713 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.119806 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:00:56.120100 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.120705 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.121526 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.121758 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.121820 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.121989 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:00:56.122100 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:00:56.122236 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.122274 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.122300 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 13:00:56.122666 25601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:00:56.124557 25601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:00:56.127192 25723 tablet_service.cc:1467] Processing CreateTablet for tablet 37c03f0a057249d38f2e16b3e9445814 (DEFAULT_TABLE table=test-table [id=35eeaee5b5824cafb5d13658c7e6c670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:56.127422 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c03f0a057249d38f2e16b3e9445814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:56.129179 25681 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Bootstrap starting.
I20260928 13:00:56.129539 25681 tablet_bootstrap.cc:654] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:56.130046 25681 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: No bootstrap required, opened a new log
I20260928 13:00:56.130095 25681 ts_tablet_manager.cc:1397] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:56.130450 25681 raft_consensus.cc:348] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.130543 25681 raft_consensus.cc:374] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:56.130599 25681 raft_consensus.cc:729] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba9579526804314a564832703be757b, State: Initialized, Role: FOLLOWER
I20260928 13:00:56.130645 25681 consensus_queue.cc:260] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.130745 25681 ts_tablet_manager.cc:1428] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 13:00:56.131351 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.126897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.11.11.66:32889)
0928 13:00:56.127090 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.127097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.127108 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:00:56.127154 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.127157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.127174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.127264 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:56.128127 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.128847 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.130110 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.130908 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.130997 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.131063 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.131224 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.131268 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260928 13:00:56.131390 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.126739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35305 (local address 127.0.0.1:59012)
0928 13:00:56.126875 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.126889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.126948 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.127455 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:56.127456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:56.127458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:56.127545 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.127546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.128850 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.128852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.130021 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.130024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.131019 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.131020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.131097 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.131124 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 13:00:56.131192 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:00:56.131257 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":68}
I20260928 13:00:56.131390 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.126739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32889 (local address 127.0.0.1:47032)
0928 13:00:56.126913 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.126922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.126967 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.127275 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:56.127276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:56.127279 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:56.127543 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.127545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.128954 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.128955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.130023 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.130026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.131019 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.131020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.131097 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.131124 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 13:00:56.131192 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:00:56.131257 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260928 13:00:56.131640 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.127081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.11.11.67:35305)
0928 13:00:56.127276 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.127281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.127293 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:00:56.127340 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.127342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.127358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.127447 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:56.127626 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.128723 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.130140 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.130932 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.131002 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.131063 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.131517 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.131567 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 13:00:56.132148 25862 tablet_service.cc:1467] Processing CreateTablet for tablet 37c03f0a057249d38f2e16b3e9445814 (DEFAULT_TABLE table=test-table [id=35eeaee5b5824cafb5d13658c7e6c670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:56.132308 25994 tablet_service.cc:1467] Processing CreateTablet for tablet 37c03f0a057249d38f2e16b3e9445814 (DEFAULT_TABLE table=test-table [id=35eeaee5b5824cafb5d13658c7e6c670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:00:56.132498 25862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c03f0a057249d38f2e16b3e9445814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:56.132632 25994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c03f0a057249d38f2e16b3e9445814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:00:56.134690 25820 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Bootstrap starting.
I20260928 13:00:56.135519 25820 tablet_bootstrap.cc:654] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:56.135813 25820 log.cc:824] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:56.136250 26070 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Bootstrap starting.
I20260928 13:00:56.136972 25820 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: No bootstrap required, opened a new log
I20260928 13:00:56.137015 26070 tablet_bootstrap.cc:654] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Neither blocks nor log segments found. Creating new log.
I20260928 13:00:56.137043 25820 ts_tablet_manager.cc:1397] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:00:56.137620 26070 log.cc:824] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:56.138427 26070 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: No bootstrap required, opened a new log
I20260928 13:00:56.138504 26070 ts_tablet_manager.cc:1397] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:00:56.139221 25820 raft_consensus.cc:348] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.139413 25820 raft_consensus.cc:374] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:56.139452 25820 raft_consensus.cc:729] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5477bd9abe3404a9e12f5fa45179735, State: Initialized, Role: FOLLOWER
I20260928 13:00:56.139624 25820 consensus_queue.cc:260] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.139958 25820 ts_tablet_manager.cc:1428] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:00:56.140014 25928 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
I20260928 13:00:56.140528 26070 raft_consensus.cc:348] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.140692 26070 raft_consensus.cc:374] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:00:56.140733 26070 raft_consensus.cc:729] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189667c4b33641578049ab49a4807f1d, State: Initialized, Role: FOLLOWER
I20260928 13:00:56.140846 26070 consensus_queue.cc:260] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.141141 26060 heartbeater.cc:503] Master 127.11.11.126:41711 was elected leader, sending a full tablet report...
I20260928 13:00:56.141399 26070 ts_tablet_manager.cc:1428] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 13:00:56.214499 25929 tablet.cc:2369] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:00:56.319584 25790 tablet.cc:2369] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:00:56.345958 26061 tablet.cc:2369] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:00:56.358953 25795 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:00:56.359130 25795 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.359526 25795 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
I20260928 13:00:56.362638 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.359825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:48281 (local address 127.11.11.67:35305)
0928 13:00:56.359904 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.359907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.359980 (+ 73us) server_negotiation.cc:407] Connection header received
0928 13:00:56.360130 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.360132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.360143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.360219 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:56.360386 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.360847 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.361869 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.362251 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.362337 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.362347 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.362527 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.362566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 13:00:56.362634 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.359819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:48309 (local address 127.11.11.66:32889)
0928 13:00:56.359870 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.359872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.359975 (+ 103us) server_negotiation.cc:407] Connection header received
0928 13:00:56.360107 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.360108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.360122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.360197 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:00:56.360345 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.360837 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.361872 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.362246 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.362340 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.362348 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.362524 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.362562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:00:56.362672 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.359720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32889 (local address 127.11.11.65:48309)
0928 13:00:56.359886 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.359896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.359998 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.360200 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:56.360201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:56.360203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:56.360263 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.360265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.360919 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.360920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.361768 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.361770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.362341 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.362342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.362414 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.362442 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 13:00:56.362507 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 13:00:56.362554 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260928 13:00:56.362663 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.359736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35305 (local address 127.11.11.65:48281)
0928 13:00:56.359897 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.359909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.359989 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.360223 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:56.360223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:56.360226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:00:56.360297 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.360299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.360934 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.360935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.361770 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:56.361772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.362337 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.362344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:56.362422 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.362440 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 13:00:56.362519 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 13:00:56.362563 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":74,"thread_start_us":87,"threads_started":1}
I20260928 13:00:56.363036 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c03f0a057249d38f2e16b3e9445814" candidate_uuid: "2ba9579526804314a564832703be757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5477bd9abe3404a9e12f5fa45179735" is_pre_election: true
I20260928 13:00:56.363265 25882 raft_consensus.cc:2454] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba9579526804314a564832703be757b in term 0.
I20260928 13:00:56.363305 26014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c03f0a057249d38f2e16b3e9445814" candidate_uuid: "2ba9579526804314a564832703be757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189667c4b33641578049ab49a4807f1d" is_pre_election: true
I20260928 13:00:56.363459 26014 raft_consensus.cc:2454] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba9579526804314a564832703be757b in term 0.
I20260928 13:00:56.363476 25676 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba9579526804314a564832703be757b, a5477bd9abe3404a9e12f5fa45179735; no voters:
I20260928 13:00:56.363624 25795 raft_consensus.cc:2781] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:00:56.363713 25795 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:00:56.363749 25795 raft_consensus.cc:3037] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:56.364488 25795 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.364645 25795 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 1 election: Requested vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
I20260928 13:00:56.364786 26014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c03f0a057249d38f2e16b3e9445814" candidate_uuid: "2ba9579526804314a564832703be757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189667c4b33641578049ab49a4807f1d"
I20260928 13:00:56.364825 25882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c03f0a057249d38f2e16b3e9445814" candidate_uuid: "2ba9579526804314a564832703be757b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5477bd9abe3404a9e12f5fa45179735"
I20260928 13:00:56.364871 26014 raft_consensus.cc:3037] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:56.364889 25882 raft_consensus.cc:3037] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:00:56.365759 25882 raft_consensus.cc:2454] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba9579526804314a564832703be757b in term 1.
I20260928 13:00:56.365759 26014 raft_consensus.cc:2454] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba9579526804314a564832703be757b in term 1.
I20260928 13:00:56.365931 25678 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 189667c4b33641578049ab49a4807f1d, 2ba9579526804314a564832703be757b; no voters:
I20260928 13:00:56.366007 25795 raft_consensus.cc:2781] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:00:56.366135 25795 raft_consensus.cc:686] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 LEADER]: Becoming Leader. State: Replica: 2ba9579526804314a564832703be757b, State: Running, Role: LEADER
I20260928 13:00:56.366227 25795 consensus_queue.cc:237] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.366967 25601 catalog_manager.cc:5495] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba9579526804314a564832703be757b (127.11.11.65). New cstate: current_term: 1 leader_uuid: "2ba9579526804314a564832703be757b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } health_report { overall_health: HEALTHY } } }
I20260928 13:00:56.415438 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.413031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.11.11.65:34397)
0928 13:00:56.413182 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:56.413184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:56.413191 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:00:56.413238 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:56.413239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:56.413263 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:56.413339 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:00:56.413482 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.413971 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.414676 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:56.414893 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:56.414952 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:56.415086 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:00:56.415192 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:00:56.415259 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:56.415294 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 13:00:56.415312 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 13:00:56.417289 25882 raft_consensus.cc:1261] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Refusing update from remote peer 2ba9579526804314a564832703be757b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:56.417289 26014 raft_consensus.cc:1261] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Refusing update from remote peer 2ba9579526804314a564832703be757b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:00:56.417724 25795 consensus_queue.cc:1035] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:56.417892 26077 consensus_queue.cc:1035] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [LEADER]: Connected to new peer: Peer: permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:00:56.420996 26081 mvcc.cc:205] Tried to move back new op lower bound from 7334299469482123264 to 7334299469276741632. Current Snapshot: MvccSnapshot[applied={T|T < 7334299469482123264}]
I20260928 13:00:56.421439 26083 mvcc.cc:205] Tried to move back new op lower bound from 7334299469482123264 to 7334299469276741632. Current Snapshot: MvccSnapshot[applied={T|T < 7334299469482123264 or (T in {7334299469482123264})}]
I20260928 13:00:56.423077 25600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:56.423197 25600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:00:56.424134 25600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Sending DeleteTablet for 3 replicas of tablet 37c03f0a057249d38f2e16b3e9445814
I20260928 13:00:56.424475 25723 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c03f0a057249d38f2e16b3e9445814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:56 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:56694
I20260928 13:00:56.424510 25862 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c03f0a057249d38f2e16b3e9445814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:56 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:47032
I20260928 13:00:56.424520 25994 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c03f0a057249d38f2e16b3e9445814 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:00:56 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:59012
I20260928 13:00:56.424784 26091 tablet_replica.cc:330] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: stopping tablet replica
I20260928 13:00:56.424787 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25639
I20260928 13:00:56.424825 26092 tablet_replica.cc:330] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: stopping tablet replica
I20260928 13:00:56.424912 26091 raft_consensus.cc:2229] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:00:56.424924 26092 raft_consensus.cc:2229] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:56.425025 26092 raft_consensus.cc:2258] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:56.425170 26091 raft_consensus.cc:2258] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:56.425220 26093 tablet_replica.cc:330] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: stopping tablet replica
I20260928 13:00:56.425318 26093 raft_consensus.cc:2229] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:00:56.425473 26093 raft_consensus.cc:2258] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:00:56.425846 26093 ts_tablet_manager.cc:1905] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:56.426223 26092 ts_tablet_manager.cc:1905] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:00:56.427831 26092 ts_tablet_manager.cc:1918] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:56.427917 26092 log.cc:1196] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/37c03f0a057249d38f2e16b3e9445814
I20260928 13:00:56.428299 26092 ts_tablet_manager.cc:1939] T 37c03f0a057249d38f2e16b3e9445814 P 189667c4b33641578049ab49a4807f1d: Deleting consensus metadata
I20260928 13:00:56.428785 25588 catalog_manager.cc:4841] TS 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): tablet 37c03f0a057249d38f2e16b3e9445814 (table test-table [id=35eeaee5b5824cafb5d13658c7e6c670]) successfully deleted
I20260928 13:00:56.428972 26093 ts_tablet_manager.cc:1918] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:00:56.429020 26093 log.cc:1196] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/37c03f0a057249d38f2e16b3e9445814
I20260928 13:00:56.429181 26093 ts_tablet_manager.cc:1939] T 37c03f0a057249d38f2e16b3e9445814 P a5477bd9abe3404a9e12f5fa45179735: Deleting consensus metadata
I20260928 13:00:56.429669 25586 catalog_manager.cc:4841] TS a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): tablet 37c03f0a057249d38f2e16b3e9445814 (table test-table [id=35eeaee5b5824cafb5d13658c7e6c670]) successfully deleted
W20260928 13:00:56.430617 25586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:34397 (error 108)
I20260928 13:00:56.431020 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25794
I20260928 13:00:56.431121 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.430984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:34397 (local address 127.0.0.1:56704)
0928 13:00:56.431034 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.431079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:34397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 13:00:56.432024 25586 catalog_manager.cc:4584] TS 2ba9579526804314a564832703be757b (127.11.11.65:34397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37c03f0a057249d38f2e16b3e9445814: Network error: Client connection negotiation failed: client connection to 127.11.11.65:34397: connect: Connection refused (error 111)
I20260928 13:00:56.435953 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25931
I20260928 13:00:56.440429 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 25568
I20260928 13:00:56.446115 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--webserver_interface=127.11.11.126
--webserver_port=33937
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:56.529354 26095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:56.529596 26095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:56.529637 26095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:56.531123 26095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:00:56.531205 26095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:56.531231 26095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:56.531265 26095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:00:56.531292 26095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:00:56.532938 26095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:56.533288 26095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:56.533527 26095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:56.538662 26101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.538609 26095 server_base.cc:1034] running on GCE node
W20260928 13:00:56.538620 26100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.538610 26103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.539108 26095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:56.539404 26095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:56.540639 26095 hybrid_clock.cc:648] HybridClock initialized: now 1790600456540630 us; error 35 us; skew 500 ppm
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:56.543306 26095 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:00:56.543784 26095 webserver.cc:460] Webserver started at http://127.11.11.126:33937/ using document root <none> and password file <none>
I20260928 13:00:56.544111 26095 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:56.544200 26095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:56.545692 26095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:56.546537 26110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.546759 26095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:00:56.546870 26095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "be3fb982b9044371a83781a9583bbd12"
format_stamp: "Formatted at 2026-09-28 13:00:54 on dist-test-slave-8sjz"
server_key: "753347420ed88df655e50eb4abab0622"
server_key_iv: "0d45995075140cb64f0893624cca2f66"
server_key_version: "encryptionkey@0"
I20260928 13:00:56.547425 26095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:56.557943 26095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:56.559998 26095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:56.560210 26095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:56.564273 26095 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41711
I20260928 13:00:56.564330 26162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41711 every 8 connection(s)
I20260928 13:00:56.564735 26095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:00:56.566262 26163 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:00:56.566913 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap starting.
I20260928 13:00:56.571414 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26095
I20260928 13:00:56.571753 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:34397
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=34457
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:56.575999 26163 log.cc:824] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:56.578363 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:00:56.578703 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12: Bootstrap complete.
I20260928 13:00:56.581085 26163 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:56.581324 26163 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Initialized, Role: FOLLOWER
I20260928 13:00:56.581492 26163 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:56.581561 26163 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:00:56.581625 26163 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:00:56.581667 26163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 2 FOLLOWER]: Advancing to term 3
I20260928 13:00:56.582824 26163 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:56.582938 26163 leader_election.cc:302] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3fb982b9044371a83781a9583bbd12; no voters:
I20260928 13:00:56.583165 26163 leader_election.cc:288] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 13:00:56.583179 26167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 13:00:56.583321 26167 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [term 3 LEADER]: Becoming Leader. State: Replica: be3fb982b9044371a83781a9583bbd12, State: Running, Role: LEADER
I20260928 13:00:56.583415 26167 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } }
I20260928 13:00:56.583462 26163 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:00:56.583716 26169 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:56.583928 26169 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:56.583873 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3fb982b9044371a83781a9583bbd12. Latest consensus state: current_term: 3 leader_uuid: "be3fb982b9044371a83781a9583bbd12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3fb982b9044371a83781a9583bbd12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41711 } } }
I20260928 13:00:56.584123 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3fb982b9044371a83781a9583bbd12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:00:56.584344 26177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:00:56.585184 26177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:00:56.585291 26177 catalog_manager.cc:1259] Loaded cluster ID: b79b502bd0a84152bfa54e903bd369c9
I20260928 13:00:56.585326 26177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 13:00:56.585616 26177 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7a9a6a99ad google::LogMessage::Fail() at ??:0
@ 0x7f7a9a6ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7a9a6a94de google::LogMessage::Flush() at ??:0
@ 0x7f7a9a6aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7aa25ca0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7aa25d38b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7a9c5770d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7a9c56c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7a9d3d06db start_thread at ??:0
@ 0x7f7a991a761f clone at ??:0
W20260928 13:00:56.661384 26165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:56.661635 26165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:56.661676 26165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:56.662883 26165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:56.662951 26165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:56.662997 26165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:00:56.664614 26165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:34397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=34457
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:56.664885 26165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:56.665120 26165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:56.670192 26200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.670280 26202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.670362 26165 server_base.cc:1034] running on GCE node
W20260928 13:00:56.670186 26199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.670722 26165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:56.671002 26165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:56.672181 26165 hybrid_clock.cc:648] HybridClock initialized: now 1790600456672166 us; error 28 us; skew 500 ppm
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:56.674995 26165 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:00:56.675493 26165 webserver.cc:460] Webserver started at http://127.11.11.65:34457/ using document root <none> and password file <none>
I20260928 13:00:56.675722 26165 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:56.675767 26165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:56.677281 26165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:56.678058 26209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.678283 26165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:56.678381 26165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "2ba9579526804314a564832703be757b"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "59a9e36e6f2e65760e27233196d931c2"
server_key_iv: "208c784ce1dbed722f84756bfee5b9ca"
server_key_version: "encryptionkey@0"
I20260928 13:00:56.678742 26165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:56.697741 26165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:56.700119 26165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:56.700321 26165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:56.700573 26165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:56.701025 26216 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:00:56.703020 26165 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:00:56.703099 26165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:00:56.703215 26165 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:00:56.703855 26165 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:00:56.703908 26165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:00:56.703987 26216 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Bootstrap starting.
I20260928 13:00:56.711201 26165 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:34397
I20260928 13:00:56.711303 26323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:34397 every 8 connection(s)
I20260928 13:00:56.711637 26165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:00:56.714974 26216 log.cc:824] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:00:56.717353 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26165
I20260928 13:00:56.717628 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:32889
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=43821
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:00:56.717741 26216 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:00:56.718037 26216 tablet_bootstrap.cc:492] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Bootstrap complete.
I20260928 13:00:56.719316 26216 ts_tablet_manager.cc:1397] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260928 13:00:56.720891 26216 raft_consensus.cc:348] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.721230 26216 raft_consensus.cc:729] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba9579526804314a564832703be757b, State: Initialized, Role: FOLLOWER
I20260928 13:00:56.721390 26216 consensus_queue.cc:260] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:56.721668 26216 ts_tablet_manager.cc:1428] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 13:00:56.801657 26326 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:00:56.712351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:60869)
0928 13:00:56.712528 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.712608 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:56.713059 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:56.801441 (+ 88382us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
W20260928 13:00:56.801991 26324 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:41711 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:41711: Client connection negotiation failed: client connection to 127.11.11.126:41711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:00:56.802385 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.802247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:50753)
0928 13:00:56.802308 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.802358 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:00:56.802755 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.802630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:56833)
0928 13:00:56.802704 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.802733 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 13:00:56.802879 26324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:00:56.807180 26328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:56.807379 26328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:56.807405 26328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:56.808655 26328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:56.808696 26328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:56.808725 26328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:00:56.810114 26328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:32889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=43821
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:56.810344 26328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:56.810525 26328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:00:56.815627 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.815622 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.815622 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.815747 26328 server_base.cc:1034] running on GCE node
I20260928 13:00:56.816155 26328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:56.816385 26328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:56.817533 26328 hybrid_clock.cc:648] HybridClock initialized: now 1790600456817523 us; error 31 us; skew 500 ppm
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:56.820204 26328 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:00:56.820695 26328 webserver.cc:460] Webserver started at http://127.11.11.66:43821/ using document root <none> and password file <none>
I20260928 13:00:56.820959 26328 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:56.821033 26328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:56.822419 26328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:56.823040 26344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.823215 26328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.823309 26328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "a5477bd9abe3404a9e12f5fa45179735"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "422ff49695a9fcd7432f1fe28af713f7"
server_key_iv: "d9a9662dd1bd935813c7a26818d7fb70"
server_key_version: "encryptionkey@0"
I20260928 13:00:56.823688 26328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:56.839386 26328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:56.841480 26328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:56.841655 26328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:56.841907 26328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:56.842224 26328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:56.842280 26328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.842325 26328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:56.842365 26328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.848181 26328 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:32889
I20260928 13:00:56.848264 26457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:32889 every 8 connection(s)
I20260928 13:00:56.848626 26328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:00:56.849726 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.849344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:34283)
0928 13:00:56.849505 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.849584 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260928 13:00:56.850121 26458 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:41711 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:41711: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
I20260928 13:00:56.850419 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.850325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:51947)
0928 13:00:56.850376 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.850401 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:00:56.850663 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.850616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:53523)
0928 13:00:56.850635 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.850652 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 13:00:56.850788 26458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:00:56.853446 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26328
I20260928 13:00:56.853718 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:35305
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:41711
--webserver_port=33289
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:00:56.936455 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:00:56.936712 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:00:56.936738 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:00:56.937974 26461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:00:56.938020 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:00:56.938048 26461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:00:56.939580 26461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:35305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=33289
--tserver_master_addrs=127.11.11.126:41711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:00:56.939851 26461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:00:56.940141 26461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:00:56.945262 26461 server_base.cc:1034] running on GCE node
W20260928 13:00:56.945284 26467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.945230 26466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:00:56.945230 26469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:00:56.945583 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:00:56.945791 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:00:56.946936 26461 hybrid_clock.cc:648] HybridClock initialized: now 1790600456946922 us; error 41 us; skew 500 ppm
Sep 28 13:00:56 dist-test-slave-8sjz krb5kdc[25022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:00:56.949476 26461 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:00:56.949956 26461 webserver.cc:460] Webserver started at http://127.11.11.67:33289/ using document root <none> and password file <none>
I20260928 13:00:56.950191 26461 fs_manager.cc:359] Metadata directory not provided
I20260928 13:00:56.950263 26461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:00:56.951650 26461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:56.952368 26476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.952560 26461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:00:56.952670 26461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "189667c4b33641578049ab49a4807f1d"
format_stamp: "Formatted at 2026-09-28 13:00:55 on dist-test-slave-8sjz"
server_key: "ffa8793103ea907ab09f4190a073c17b"
server_key_iv: "24458ca1875d7ae4aba8a934671a84a9"
server_key_version: "encryptionkey@0"
I20260928 13:00:56.953047 26461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:00:56.971866 26461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:00:56.973937 26461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:00:56.974118 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:00:56.974387 26461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:00:56.974721 26461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:00:56.974777 26461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.974818 26461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:00:56.974848 26461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:00:56.981055 26461 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35305
I20260928 13:00:56.981099 26589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35305 every 8 connection(s)
I20260928 13:00:56.981488 26461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:00:56.982733 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.982298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:47189)
0928 13:00:56.982459 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.982540 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20260928 13:00:56.983134 26590 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:41711 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:41711: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
I20260928 13:00:56.983415 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.983322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:56579)
0928 13:00:56.983369 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.983397 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:00:56.983676 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:56.983606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:37347)
0928 13:00:56.983643 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:56.983663 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 13:00:56.983791 26590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:00:56.988843 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26461
I20260928 13:00:57.803545 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:57.803291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:33497)
0928 13:00:57.803433 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:57.803478 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 13:00:57.851331 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:57.851100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:42135)
0928 13:00:57.851242 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:57.851281 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260928 13:00:57.984453 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:57.984225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:46685)
0928 13:00:57.984353 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:57.984396 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 13:00:58.451370 26597 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:00:58.451496 26597 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:00:58.451783 26597 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
Sep 28 13:00:58 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.66@KRBTEST.COM
Sep 28 13:00:58 dist-test-slave-8sjz krb5kdc[25022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600456, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.67@KRBTEST.COM
I20260928 13:00:58.464983 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.452140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:41137 (local address 127.11.11.66:32889)
0928 13:00:58.452308 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:58.452321 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:58.452359 (+ 38us) server_negotiation.cc:407] Connection header received
0928 13:00:58.452960 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:58.452964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:58.453028 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:58.453196 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:58.454282 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.455724 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.457313 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.457954 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.458074 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:58.461024 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:58.461028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:58.461028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:58.461040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:58.462949 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463670 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:58.463675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:58.463675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:58.463770 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:58.464082 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:58.464083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:58.464083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:58.464347 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:58.464446 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:58.464723 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 13:00:58.464863 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 13:00:58.464998 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.451987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:32889 (local address 127.11.11.65:41137)
0928 13:00:58.452165 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:58.452174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:58.452212 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:58.453268 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:58.453272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:58.453846 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:58.454143 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:58.454152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.455853 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.455854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:58.457215 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:58.457219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.458087 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.458088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:58.458256 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:58.458943 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:58.458948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:58.460878 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:58.463064 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:58.463075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:58.463574 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:58.463857 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:58.463858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:58.463974 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:58.464427 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:58.464428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:58.464552 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 13:00:58.464654 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 13:00:58.464856 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":9,"spinlock_wait_cycles":16640,"thread_start_us":78,"threads_started":1}
I20260928 13:00:58.465019 26601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.452140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:57303 (local address 127.11.11.67:35305)
0928 13:00:58.452308 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 13:00:58.452321 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:00:58.452359 (+ 38us) server_negotiation.cc:407] Connection header received
0928 13:00:58.452971 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:00:58.452974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:00:58.453028 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:00:58.453196 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:00:58.454282 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.455715 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.457312 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.457954 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.458074 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:58.461357 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:00:58.461362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:00:58.461362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:00:58.461373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:00:58.463002 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463667 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:58.463671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:58.463671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:58.463776 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:00:58.464062 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:00:58.464063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:00:58.464064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:00:58.464333 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:00:58.464431 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:00:58.464742 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 13:00:58.464901 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260928 13:00:58.465222 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.451987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35305 (local address 127.11.11.65:57303)
0928 13:00:58.452131 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:58.452143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:00:58.452212 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:00:58.453268 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:00:58.453272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:00:58.453847 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:00:58.454143 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:58.454162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.455841 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.455842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:58.457220 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:00:58.457222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:00:58.458089 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:00:58.458090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:00:58.458265 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:00:58.458921 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:00:58.458932 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:00:58.461241 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:00:58.463106 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463120 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:58.463120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:58.463583 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:58.463849 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:00:58.463850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:00:58.463851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:00:58.463972 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:00:58.464411 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:00:58.464418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:00:58.464552 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 13:00:58.464655 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 13:00:58.464939 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":329,"thread_start_us":78,"threads_started":1}
W20260928 13:00:58.466307 26213 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
W20260928 13:00:58.466378 26211 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
I20260928 13:00:58.466406 26213 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ba9579526804314a564832703be757b; no voters: 189667c4b33641578049ab49a4807f1d, a5477bd9abe3404a9e12f5fa45179735
I20260928 13:00:58.466516 26597 raft_consensus.cc:2726] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:00:58.804260 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.804121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:35313)
0928 13:00:58.804177 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:58.804215 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:00:58.852129 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.851840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:37719)
0928 13:00:58.851994 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:58.852076 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 13:00:58.985085 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:58.984871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:54605)
0928 13:00:58.985000 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:58.985040 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260928 13:00:59.804993 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:59.804756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:58399)
0928 13:00:59.804894 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:59.804936 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260928 13:00:59.852771 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:59.852582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:36297)
0928 13:00:59.852695 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:59.852727 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260928 13:00:59.985864 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:00:59.985623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:33961)
0928 13:00:59.985774 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:00:59.985819 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 13:01:00.691532 26607 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:00.691673 26607 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:01:00.691891 26607 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
W20260928 13:01:00.692425 26211 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
W20260928 13:01:00.692500 26213 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
I20260928 13:01:00.692545 26213 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ba9579526804314a564832703be757b; no voters: 189667c4b33641578049ab49a4807f1d, a5477bd9abe3404a9e12f5fa45179735
I20260928 13:01:00.692644 26607 raft_consensus.cc:2726] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:00.805881 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:00.805597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:58915)
0928 13:01:00.805765 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:00.805812 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260928 13:01:00.853546 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:00.853258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:40115)
0928 13:01:00.853431 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:00.853483 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260928 13:01:00.986694 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:00.986455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:35601)
0928 13:01:00.986597 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:00.986645 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260928 13:01:01.806594 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:01.806389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:44047)
0928 13:01:01.806511 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:01.806545 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 13:01:01.854333 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:01.854112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:55537)
0928 13:01:01.854236 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:01.854279 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 13:01:01.987447 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:01.987225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:58311)
0928 13:01:01.987362 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:01.987399 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 13:01:02.457854 26614 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:02.458020 26614 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:01:02.458259 26614 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
W20260928 13:01:02.458746 26211 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
W20260928 13:01:02.458837 26213 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
I20260928 13:01:02.458882 26213 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ba9579526804314a564832703be757b; no voters: 189667c4b33641578049ab49a4807f1d, a5477bd9abe3404a9e12f5fa45179735
I20260928 13:01:02.458981 26614 raft_consensus.cc:2726] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:02.807421 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:02.807167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:42149)
0928 13:01:02.807325 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:02.807371 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260928 13:01:02.855064 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:02.854855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:48457)
0928 13:01:02.854980 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:02.855017 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 13:01:02.988348 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:02.988001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:41191)
0928 13:01:02.988225 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:02.988298 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":159,"threads_started":1}
I20260928 13:01:03.808400 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:03.808104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:53603)
0928 13:01:03.808286 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:03.808343 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260928 13:01:03.855814 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:03.855573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:48785)
0928 13:01:03.855726 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:03.855766 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260928 13:01:03.989228 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:03.988941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:50153)
0928 13:01:03.989113 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:03.989169 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260928 13:01:04.809307 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:04.809010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:49281)
0928 13:01:04.809176 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:04.809233 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260928 13:01:04.856755 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:04.856447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:54445)
0928 13:01:04.856597 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:04.856705 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260928 13:01:04.990069 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:04.989833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:42475)
0928 13:01:04.989980 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:04.990019 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260928 13:01:05.106684 26624 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:05.106822 26624 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:01:05.107201 26624 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
W20260928 13:01:05.107645 26211 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
W20260928 13:01:05.107756 26213 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
I20260928 13:01:05.107818 26213 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ba9579526804314a564832703be757b; no voters: 189667c4b33641578049ab49a4807f1d, a5477bd9abe3404a9e12f5fa45179735
I20260928 13:01:05.107935 26624 raft_consensus.cc:2726] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:05.810015 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:05.809779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:49593)
0928 13:01:05.809927 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:05.809966 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260928 13:01:05.857584 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:05.857337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:54381)
0928 13:01:05.857483 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:05.857532 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260928 13:01:05.990721 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:05.990477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:48907)
0928 13:01:05.990632 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:05.990671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260928 13:01:06.810922 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:06.810672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:54027)
0928 13:01:06.810825 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:06.810871 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260928 13:01:06.858467 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:06.858126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:54029)
0928 13:01:06.858324 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:06.858402 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260928 13:01:06.991645 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:06.991413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:41651)
0928 13:01:06.991547 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:06.991595 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260928 13:01:07.351553 26631 raft_consensus.cc:482] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:07.351663 26631 raft_consensus.cc:504] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "189667c4b33641578049ab49a4807f1d" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35305 } } peers { permanent_uuid: "a5477bd9abe3404a9e12f5fa45179735" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 32889 } } peers { permanent_uuid: "2ba9579526804314a564832703be757b" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 34397 } }
I20260928 13:01:07.351931 26631 leader_election.cc:288] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305), a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889)
W20260928 13:01:07.352370 26211 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5477bd9abe3404a9e12f5fa45179735 (127.11.11.66:32889): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
W20260928 13:01:07.352504 26213 leader_election.cc:341] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 189667c4b33641578049ab49a4807f1d (127.11.11.67:35305): Not found: Tablet not found: 37c03f0a057249d38f2e16b3e9445814
I20260928 13:01:07.352566 26213 leader_election.cc:302] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ba9579526804314a564832703be757b; no voters: 189667c4b33641578049ab49a4807f1d, a5477bd9abe3404a9e12f5fa45179735
I20260928 13:01:07.352700 26631 raft_consensus.cc:2726] T 37c03f0a057249d38f2e16b3e9445814 P 2ba9579526804314a564832703be757b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:07.811762 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:07.811509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:41045)
0928 13:01:07.811660 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:07.811711 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260928 13:01:07.859233 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:07.858984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:43791)
0928 13:01:07.859133 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:07.859178 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260928 13:01:07.992547 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:07.992274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:60633)
0928 13:01:07.992448 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:07.992498 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 13:01:08.812634 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:08.812388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:37585)
0928 13:01:08.812536 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:08.812585 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 13:01:08.860026 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:08.859764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:37209)
0928 13:01:08.859927 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:08.859978 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260928 13:01:08.993445 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:08.993168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:60539)
0928 13:01:08.993332 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:08.993381 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260928 13:01:09.813382 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:09.813121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:45715)
0928 13:01:09.813273 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:09.813330 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260928 13:01:09.860860 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:09.860627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:49263)
0928 13:01:09.860761 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:09.860809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 13:01:09.994253 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:09.993975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:36579)
0928 13:01:09.994160 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:09.994203 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260928 13:01:10.814217 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:10.813976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:39947)
0928 13:01:10.814122 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:10.814165 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260928 13:01:10.861732 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:10.861485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:50517)
0928 13:01:10.861635 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:10.861683 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 13:01:10.995092 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:10.994820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:50961)
0928 13:01:10.994989 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:10.995038 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260928 13:01:11.815068 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:11.814803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.65:55487)
0928 13:01:11.814971 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:11.815021 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260928 13:01:11.862596 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:11.862353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.66:57533)
0928 13:01:11.862505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:11.862548 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260928 13:01:11.996014 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:11.995741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41711 (local address 127.11.11.67:44049)
0928 13:01:11.995917 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:11.995968 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:41711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260928 13:01:11.997258 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26165
I20260928 13:01:12.002166 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26328
I20260928 13:01:12.006839 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26461
2026-09-28T13:01:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18415 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:01:12 dist-test-slave-8sjz krb5kdc[26649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:01:12 dist-test-slave-8sjz krb5kdc[26649](info): set up 2 sockets
Sep 28 13:01:12 dist-test-slave-8sjz krb5kdc[26649](info): commencing operation
krb5kdc: starting...
W20260928 13:01:13.097244 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:01:13.223690 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:13.310665 26662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:13.310933 26662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:13.310974 26662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:13.312178 26662 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:01:13.312255 26662 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:01:13.312275 26662 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:01:13.312297 26662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:01:13.312326 26662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:13.312359 26662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:13.312383 26662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:01:13.312414 26662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:01:13.313963 26662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:13.314289 26662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:13.314543 26662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:13.319938 26667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:13.319960 26668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.320006 26662 server_base.cc:1034] running on GCE node
W20260928 13:01:13.319998 26670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.320555 26662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:13.320901 26662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:13.322090 26662 hybrid_clock.cc:648] HybridClock initialized: now 1790600473322067 us; error 42 us; skew 500 ppm
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:13.324673 26662 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:01:13.325088 26662 webserver.cc:460] Webserver started at http://127.11.11.126:44829/ using document root <none> and password file <none>
I20260928 13:01:13.325279 26662 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:13.325320 26662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:13.325412 26662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:01:13.326460 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "475a2fc8197b4b3595b6e8b16ceddbbd"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "2c006f75265d3f6a4434ba23235ab428"
server_key_iv: "35b757a48427f12f475899d3bcc28e03"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.326772 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "475a2fc8197b4b3595b6e8b16ceddbbd"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "2c006f75265d3f6a4434ba23235ab428"
server_key_iv: "35b757a48427f12f475899d3bcc28e03"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.328159 26662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:13.328996 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.329166 26662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:13.329243 26662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "475a2fc8197b4b3595b6e8b16ceddbbd"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "2c006f75265d3f6a4434ba23235ab428"
server_key_iv: "35b757a48427f12f475899d3bcc28e03"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.329325 26662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:13.353062 26662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:13.355209 26662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:13.355386 26662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:13.359704 26662 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46085
I20260928 13:01:13.359782 26729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46085 every 8 connection(s)
I20260928 13:01:13.360164 26662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:01:13.361143 26730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:13.363528 26730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap starting.
I20260928 13:01:13.364183 26730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:13.364447 26730 log.cc:824] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:13.365116 26730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: No bootstrap required, opened a new log
I20260928 13:01:13.366688 26730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:13.366830 26730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:13.366853 26730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Initialized, Role: FOLLOWER
I20260928 13:01:13.366962 26730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:13.367048 26730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:01:13.367084 26730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:01:13.367138 26730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:13.367717 26730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:13.367848 26730 leader_election.cc:302] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475a2fc8197b4b3595b6e8b16ceddbbd; no voters:
I20260928 13:01:13.368083 26730 leader_election.cc:288] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:01:13.368124 26733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:01:13.368330 26733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 LEADER]: Becoming Leader. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Running, Role: LEADER
I20260928 13:01:13.368357 26730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:01:13.368428 26733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:13.369040 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26662
I20260928 13:01:13.369071 26734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:13.369148 26734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:13.369155 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:01:13.369387 11309 external_mini_cluster.cc:1428] Setting key 062a455f0c7715406e1e900909709e02
I20260928 13:01:13.369480 26735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475a2fc8197b4b3595b6e8b16ceddbbd. Latest consensus state: current_term: 1 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:13.369560 26735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:13.369683 26742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:01:13.370306 26742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:01:13.371910 26742 catalog_manager.cc:1347] Generated new cluster ID: d44db42b9701494192edaa477a9215f7
I20260928 13:01:13.371961 26742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:13.380127 26742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:01:13.380710 26742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:01:13.381573 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.370109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.11.11.126:46085)
0928 13:01:13.370263 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.370273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.370308 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:01:13.370432 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.370439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.370497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.370637 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:13.371387 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.372807 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.373867 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.374515 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.374631 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.377573 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:13.377577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:13.377577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:13.377589 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:13.379571 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.380021 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.380024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.380025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.380163 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.380426 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.380427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.380427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.380771 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:13.380949 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.381062 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.381329 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260928 13:01:13.385190 26742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Generated new TSK 0
I20260928 13:01:13.385386 26742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:01:13.413007 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:13.497246 26756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:13.497498 26756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:13.497540 26756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:13.498739 26756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:13.498811 26756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:13.498862 26756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:01:13.500449 26756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:13.500727 26756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:13.500980 26756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:13.506182 26761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:13.506131 26762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:13.506134 26764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.506415 26756 server_base.cc:1034] running on GCE node
I20260928 13:01:13.506611 26756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:13.506855 26756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:13.508013 26756 hybrid_clock.cc:648] HybridClock initialized: now 1790600473507997 us; error 29 us; skew 500 ppm
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:13.510710 26756 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:01:13.511126 26756 webserver.cc:460] Webserver started at http://127.11.11.65:32957/ using document root <none> and password file <none>
I20260928 13:01:13.511336 26756 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:13.511379 26756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:13.511471 26756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:01:13.512604 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "40852c363525ce642efdf1aa63518273"
server_key_iv: "b2a8f230d244ee5081f2a59f3152052c"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.512934 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "40852c363525ce642efdf1aa63518273"
server_key_iv: "b2a8f230d244ee5081f2a59f3152052c"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.514204 26756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:13.514922 26771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.515113 26756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:01:13.515219 26756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "40852c363525ce642efdf1aa63518273"
server_key_iv: "b2a8f230d244ee5081f2a59f3152052c"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.515297 26756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:13.526255 26756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:13.528846 26756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:13.529024 26756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:13.529289 26756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:13.529623 26756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:13.529678 26756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.529713 26756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:13.529757 26756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.535638 26756 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38537
I20260928 13:01:13.535722 26884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38537 every 8 connection(s)
I20260928 13:01:13.536094 26756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:01:13.538691 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26756
I20260928 13:01:13.538781 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:01:13.539022 11309 external_mini_cluster.cc:1428] Setting key 6aaf061c1f0fe44e04d7db80497ba859
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:13.547443 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.536913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:47179)
0928 13:01:13.537094 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.537132 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.537662 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.537927 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.537931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.538558 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:13.538875 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.538882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.539764 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.539765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.541279 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.541282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.541846 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.541848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.541967 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.542579 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:13.542588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:13.544111 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:13.546031 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.546038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.546040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.546284 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.546509 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.546510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.546510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.546624 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.546917 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:13.546921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:13.547051 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.547136 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.547268 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 13:01:13.547438 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.536929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47179 (local address 127.11.11.126:46085)
0928 13:01:13.537002 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.537007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.537229 (+ 222us) server_negotiation.cc:407] Connection header received
0928 13:01:13.537781 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.537782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.537794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.537852 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:13.538990 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.539624 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.541387 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.541725 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.541815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.544216 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:13.544218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:13.544218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:13.544227 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:13.545936 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.546365 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.546367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.546367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.546436 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.546692 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.546693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.546693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.546849 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:13.546925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.547191 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.547289 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 13:01:13.548115 26885 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:13.548228 26885 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:13.548451 26885 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:13.549026 26694 ts_manager.cc:194] Registered new tserver with Master: fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537)
I20260928 13:01:13.549760 26694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:47179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:01:13.567312 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:13.650571 26890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:13.650811 26890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:13.650836 26890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:13.652099 26890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:13.652143 26890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:13.652171 26890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:01:13.653702 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:13.653986 26890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:13.654233 26890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:13.660635 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:13.660607 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.660681 26890 server_base.cc:1034] running on GCE node
W20260928 13:01:13.660619 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.661074 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:13.661306 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:13.662456 26890 hybrid_clock.cc:648] HybridClock initialized: now 1790600473662446 us; error 33 us; skew 500 ppm
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:13.665247 26890 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:01:13.665784 26890 webserver.cc:460] Webserver started at http://127.11.11.66:39103/ using document root <none> and password file <none>
I20260928 13:01:13.666038 26890 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:13.666085 26890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:13.666258 26890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:01:13.667402 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "39664385a0f1451a90a11ab2076ef5ef"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "6ea0538de03f396a0e691b44eefc70e2"
server_key_iv: "5ba42c19cabd6bd9d8b3bb11c69ff523"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.667785 26890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "39664385a0f1451a90a11ab2076ef5ef"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "6ea0538de03f396a0e691b44eefc70e2"
server_key_iv: "5ba42c19cabd6bd9d8b3bb11c69ff523"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.669409 26890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:01:13.670565 26905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.670888 26890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:13.671002 26890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "39664385a0f1451a90a11ab2076ef5ef"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "6ea0538de03f396a0e691b44eefc70e2"
server_key_iv: "5ba42c19cabd6bd9d8b3bb11c69ff523"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.671082 26890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:13.704571 26890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:13.706969 26890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:13.707141 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:13.707408 26890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:13.707724 26890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:13.707779 26890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.707818 26890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:13.707864 26890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.714428 26890 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:39765
I20260928 13:01:13.714485 27018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:39765 every 8 connection(s)
I20260928 13:01:13.714845 26890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:13.723057 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 26890
I20260928 13:01:13.723150 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:01:13.723361 11309 external_mini_cluster.cc:1428] Setting key 448a79a7ca1513402443316ec4d65ac8
I20260928 13:01:13.725106 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.715656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:53593 (local address 127.11.11.126:46085)
0928 13:01:13.715725 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.715728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.715943 (+ 215us) server_negotiation.cc:407] Connection header received
0928 13:01:13.716516 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.716517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.716527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.716576 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:13.717533 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.718035 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.719096 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.719381 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.719437 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.721616 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:13.721618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:13.721618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:13.721626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:13.723556 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.724013 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.724014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.724015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.724109 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.724434 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.724435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.724435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.724552 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:13.724608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.724957 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.725034 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:01:13.725436 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.715633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:53593)
0928 13:01:13.715811 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.715850 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.716423 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.716639 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.716643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.717216 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:13.717430 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.717435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.718157 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.718158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.719016 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.719018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.719457 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.719457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.719562 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.720072 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:13.720084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:13.721510 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:13.723656 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.723665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.723667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.723935 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.724194 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.724195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.724195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.724362 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.724619 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:13.724626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:13.724810 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.724932 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.725120 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260928 13:01:13.726042 27019 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:13.726158 27019 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:13.726370 27019 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:13.726819 26694 ts_manager.cc:194] Registered new tserver with Master: 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765)
I20260928 13:01:13.727451 26694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:53593
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:01:13.750888 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:13.834189 27024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:13.834405 27024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:13.834429 27024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:13.835647 27024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:13.835693 27024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:13.835726 27024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:01:13.837289 27024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:13.837551 27024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:13.837800 27024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:13.842808 27030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:13.842808 27032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.843263 27024 server_base.cc:1034] running on GCE node
W20260928 13:01:13.842813 27029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:13.843545 27024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:13.843779 27024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:13.844934 27024 hybrid_clock.cc:648] HybridClock initialized: now 1790600473844920 us; error 28 us; skew 500 ppm
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:13.847690 27024 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:01:13.848220 27024 webserver.cc:460] Webserver started at http://127.11.11.67:33729/ using document root <none> and password file <none>
I20260928 13:01:13.848465 27024 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:13.848536 27024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:13.848719 27024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:01:13.849799 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "69ee8725ef1a4a5898165a56837c7b18"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "8cf9f1c1dc2781e111826f3a05901b14"
server_key_iv: "b6d8d885c3f1b45263ae0e4e4b6939e0"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.850189 27024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "69ee8725ef1a4a5898165a56837c7b18"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "8cf9f1c1dc2781e111826f3a05901b14"
server_key_iv: "b6d8d885c3f1b45263ae0e4e4b6939e0"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.851584 27024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:01:13.852420 27039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.852617 27024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:01:13.852725 27024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "69ee8725ef1a4a5898165a56837c7b18"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "8cf9f1c1dc2781e111826f3a05901b14"
server_key_iv: "b6d8d885c3f1b45263ae0e4e4b6939e0"
server_key_version: "encryptionkey@0"
I20260928 13:01:13.852803 27024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:13.863575 27024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:13.865589 27024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:13.865778 27024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:13.866025 27024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:13.866382 27024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:13.866441 27024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.866483 27024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:13.866531 27024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:13.872844 27024 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:42743
I20260928 13:01:13.872893 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:42743 every 8 connection(s)
I20260928 13:01:13.873210 27024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:01:13.876121 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27024
I20260928 13:01:13.876238 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:01:13.876420 11309 external_mini_cluster.cc:1428] Setting key a6d3dbebf60dabcb3ba845102fba313e
Sep 28 13:01:13 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:13.882841 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.873886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:37233 (local address 127.11.11.126:46085)
0928 13:01:13.873942 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.873944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.874187 (+ 243us) server_negotiation.cc:407] Connection header received
0928 13:01:13.874669 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.874670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.874682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.874736 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:13.876082 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.876853 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.877885 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.878101 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.878189 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.880412 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:13.880413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:13.880414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:13.880421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:13.881611 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.881990 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.881991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.881991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.882043 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.882306 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.882307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.882307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.882415 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:13.882466 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.882702 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.882767 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:01:13.883008 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.873840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:37233)
0928 13:01:13.874026 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.874089 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.874584 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.874790 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.874793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.875606 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:13.875942 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.875948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.876948 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.876949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.877770 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.877772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.878199 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.878200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.878342 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.878842 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:13.878851 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:13.880327 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:13.881686 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.881692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.881694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.881922 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.882117 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:13.882118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:13.882118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:13.882239 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:13.882471 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:13.882476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:13.882591 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.882673 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.882795 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260928 13:01:13.883513 27153 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:13.883625 27153 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:13.883800 27153 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:13.884240 26694 ts_manager.cc:194] Registered new tserver with Master: 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
I20260928 13:01:13.884619 26694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:37233
I20260928 13:01:13.888304 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:01:13.895617 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.889609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.11.11.126:46085)
0928 13:01:13.889648 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.889650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.889764 (+ 114us) server_negotiation.cc:407] Connection header received
0928 13:01:13.889806 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.889806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.889814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.889871 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:13.890445 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.890922 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.891632 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.891830 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.891889 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.893289 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:13.893290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:13.893291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:13.893297 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:13.894528 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.894843 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.894844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.894844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.894906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:13.895097 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:13.895097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:13.895097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:13.895277 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:13.895356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.895479 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.895552 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 13:01:13.897754 26694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:01:13.899430 26694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:01:13.906580 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.902034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.11.11.67:42743)
0928 13:01:13.902636 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.902640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.902780 (+ 140us) server_negotiation.cc:407] Connection header received
0928 13:01:13.902821 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.902823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.902839 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.902945 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:13.903046 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.904204 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.905445 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.906106 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.906247 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.906304 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.906447 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.906487 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":101,"threads_started":1}
I20260928 13:01:13.906589 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.901935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:42743 (local address 127.0.0.1:40464)
0928 13:01:13.902263 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.902313 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.902386 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.902950 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.902951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.902952 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:13.902989 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.902990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.904303 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.904304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.905355 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.905357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.906251 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.906251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.906328 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.906343 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.906416 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.906469 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":47,"threads_started":1}
I20260928 13:01:13.906592 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.902123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38537 (local address 127.0.0.1:47746)
0928 13:01:13.902435 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.902445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.902501 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.902836 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.902837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.902839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:13.902945 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.902947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.903886 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.903887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.905231 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.905233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.906282 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.906283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.906352 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.906358 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.906441 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.906487 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":20,"thread_start_us":78,"threads_started":1}
I20260928 13:01:13.907155 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.902240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.11.11.65:38537)
0928 13:01:13.902434 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.902440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.902492 (+ 52us) server_negotiation.cc:407] Connection header received
0928 13:01:13.902571 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.902573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.902586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.902680 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:13.903032 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.903778 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.905319 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.906099 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.906177 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.906247 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.906828 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.907065 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260928 13:01:13.907574 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.901935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39765 (local address 127.0.0.1:41776)
0928 13:01:13.902288 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.902320 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.902387 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.902814 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.902818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.902820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:13.902944 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.902947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.905442 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.905442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.906089 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.906090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.907373 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.907374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.907453 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.907456 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.907498 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.907528 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260928 13:01:13.907662 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.902151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.11.11.66:39765)
0928 13:01:13.902537 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.902543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.902554 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:01:13.902622 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.902625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.902641 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.902714 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:13.903036 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.904193 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.906177 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.907222 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.907286 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.907339 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.907539 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.907587 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":77,"threads_started":1}
I20260928 13:01:13.907948 27087 tablet_service.cc:1467] Processing CreateTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 (DEFAULT_TABLE table=test-table [id=7186f29cf08a403f9380760dbb348d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:13.908377 27087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a3d20c106a4ce1969b963c0cefe865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:13.908483 26953 tablet_service.cc:1467] Processing CreateTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 (DEFAULT_TABLE table=test-table [id=7186f29cf08a403f9380760dbb348d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:13.908749 26819 tablet_service.cc:1467] Processing CreateTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 (DEFAULT_TABLE table=test-table [id=7186f29cf08a403f9380760dbb348d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:13.908866 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a3d20c106a4ce1969b963c0cefe865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:13.909011 26819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a3d20c106a4ce1969b963c0cefe865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:13.911693 27170 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap starting.
I20260928 13:01:13.911921 27171 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Bootstrap starting.
I20260928 13:01:13.912411 27170 tablet_bootstrap.cc:654] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:13.912814 27171 tablet_bootstrap.cc:654] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:13.913053 27170 log.cc:824] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:13.913398 27171 log.cc:824] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:13.913771 27172 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Bootstrap starting.
I20260928 13:01:13.913956 27170 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: No bootstrap required, opened a new log
I20260928 13:01:13.914038 27170 ts_tablet_manager.cc:1397] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:01:13.914391 27172 tablet_bootstrap.cc:654] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:13.914927 27172 log.cc:824] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:13.915683 27172 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: No bootstrap required, opened a new log
I20260928 13:01:13.915753 27172 ts_tablet_manager.cc:1397] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:01:13.916283 27170 raft_consensus.cc:348] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.916478 27170 raft_consensus.cc:374] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:13.916517 27170 raft_consensus.cc:729] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcfbe72b22a49d4b6346f8f0828653c, State: Initialized, Role: FOLLOWER
I20260928 13:01:13.916646 27170 consensus_queue.cc:260] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.916708 27171 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: No bootstrap required, opened a new log
I20260928 13:01:13.916770 27171 ts_tablet_manager.cc:1397] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 13:01:13.916939 27170 ts_tablet_manager.cc:1428] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 13:01:13.917097 26885 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:13.918648 27171 raft_consensus.cc:348] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.918816 27171 raft_consensus.cc:374] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:13.918865 27171 raft_consensus.cc:729] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ee8725ef1a4a5898165a56837c7b18, State: Initialized, Role: FOLLOWER
I20260928 13:01:13.918984 27171 consensus_queue.cc:260] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.919263 27171 ts_tablet_manager.cc:1428] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:01:13.919297 27153 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:13.920758 27172 raft_consensus.cc:348] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.920922 27172 raft_consensus.cc:374] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:13.920969 27172 raft_consensus.cc:729] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39664385a0f1451a90a11ab2076ef5ef, State: Initialized, Role: FOLLOWER
I20260928 13:01:13.921090 27172 consensus_queue.cc:260] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.921383 27172 ts_tablet_manager.cc:1428] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260928 13:01:13.921454 27019 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:13.942202 27176 raft_consensus.cc:482] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:13.942350 27176 raft_consensus.cc:504] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.942691 27176 leader_election.cc:288] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
I20260928 13:01:13.945675 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.942935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39765 (local address 127.11.11.65:58811)
0928 13:01:13.942993 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.943006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.943077 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.943250 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.943252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.943255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:13.943347 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.943349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.944007 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.944009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.944793 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.944795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.945391 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.945392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.945480 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.945484 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.945552 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.945599 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:01:13.945675 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.943040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:58811 (local address 127.11.11.66:39765)
0928 13:01:13.943085 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.943088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.943106 (+ 18us) server_negotiation.cc:407] Connection header received
0928 13:01:13.943134 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.943145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.943153 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.943227 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:13.943429 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.943913 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.944901 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.945297 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.945383 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.945393 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.945558 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.945600 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:01:13.945971 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.943021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:51775 (local address 127.11.11.67:42743)
0928 13:01:13.943098 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.943100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.943404 (+ 304us) server_negotiation.cc:407] Connection header received
0928 13:01:13.943479 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.943481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.943490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.943572 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:13.943738 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.944273 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.945195 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.945572 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.945668 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.945676 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.945856 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.945902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:01:13.946084 26973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a3d20c106a4ce1969b963c0cefe865" candidate_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39664385a0f1451a90a11ab2076ef5ef" is_pre_election: true
I20260928 13:01:13.946193 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.942935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:42743 (local address 127.11.11.65:51775)
0928 13:01:13.943325 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.943334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:13.943415 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:13.943577 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:13.943578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:13.943579 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:13.943662 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.943664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.944357 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.944358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.945102 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:13.945104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.945671 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.945672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:13.945748 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.945753 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:01:13.946069 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 13:01:13.946119 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":195,"thread_start_us":127,"threads_started":1}
I20260928 13:01:13.946262 26973 raft_consensus.cc:2454] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdcfbe72b22a49d4b6346f8f0828653c in term 0.
I20260928 13:01:13.946503 27107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a3d20c106a4ce1969b963c0cefe865" candidate_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ee8725ef1a4a5898165a56837c7b18" is_pre_election: true
I20260928 13:01:13.946556 26773 leader_election.cc:302] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39664385a0f1451a90a11ab2076ef5ef, fdcfbe72b22a49d4b6346f8f0828653c; no voters:
I20260928 13:01:13.946668 27107 raft_consensus.cc:2454] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdcfbe72b22a49d4b6346f8f0828653c in term 0.
I20260928 13:01:13.946712 27176 raft_consensus.cc:2781] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:01:13.946782 27176 raft_consensus.cc:482] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:01:13.946815 27176 raft_consensus.cc:3037] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:13.947538 27176 raft_consensus.cc:504] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.947686 27176 leader_election.cc:288] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 1 election: Requested vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
I20260928 13:01:13.947834 26973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a3d20c106a4ce1969b963c0cefe865" candidate_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39664385a0f1451a90a11ab2076ef5ef"
I20260928 13:01:13.947916 26973 raft_consensus.cc:3037] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:13.948100 27107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a3d20c106a4ce1969b963c0cefe865" candidate_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ee8725ef1a4a5898165a56837c7b18"
I20260928 13:01:13.948199 27107 raft_consensus.cc:3037] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:13.948794 26973 raft_consensus.cc:2454] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcfbe72b22a49d4b6346f8f0828653c in term 1.
I20260928 13:01:13.949015 26773 leader_election.cc:302] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39664385a0f1451a90a11ab2076ef5ef, fdcfbe72b22a49d4b6346f8f0828653c; no voters:
I20260928 13:01:13.949088 27107 raft_consensus.cc:2454] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcfbe72b22a49d4b6346f8f0828653c in term 1.
I20260928 13:01:13.949108 27176 raft_consensus.cc:2781] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:01:13.949283 27176 raft_consensus.cc:686] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 LEADER]: Becoming Leader. State: Replica: fdcfbe72b22a49d4b6346f8f0828653c, State: Running, Role: LEADER
I20260928 13:01:13.949389 27176 consensus_queue.cc:237] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:13.950237 26694 catalog_manager.cc:5495] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c reported cstate change: term changed from 0 to 1, leader changed from <none> to fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65). New cstate: current_term: 1 leader_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } health_report { overall_health: HEALTHY } } }
I20260928 13:01:13.964753 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.962130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.11.11.65:38537)
0928 13:01:13.962250 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:13.962253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:13.962260 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:01:13.962296 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:13.962297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:13.962318 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:13.962396 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:01:13.962557 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.963040 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.963841 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:13.964094 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:13.964191 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:13.964375 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:01:13.964491 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:01:13.964588 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:13.964608 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 13:01:13.964630 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 13:01:13.965399 27020 tablet.cc:2369] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:01:13.966699 26973 raft_consensus.cc:1261] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Refusing update from remote peer fdcfbe72b22a49d4b6346f8f0828653c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:01:13.966722 27107 raft_consensus.cc:1261] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Refusing update from remote peer fdcfbe72b22a49d4b6346f8f0828653c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:01:13.967109 27180 consensus_queue.cc:1035] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:01:13.967257 27176 consensus_queue.cc:1035] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:01:13.969858 27183 mvcc.cc:205] Tried to move back new op lower bound from 7334299541364719616 to 7334299541297250304. Current Snapshot: MvccSnapshot[applied={T|T < 7334299541364719616}]
I20260928 13:01:13.969980 27185 mvcc.cc:205] Tried to move back new op lower bound from 7334299541364719616 to 7334299541297250304. Current Snapshot: MvccSnapshot[applied={T|T < 7334299541364719616}]
I20260928 13:01:13.972781 26693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:01:13.972899 26693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:01:13.973793 26693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Sending DeleteTablet for 3 replicas of tablet 02a3d20c106a4ce1969b963c0cefe865
I20260928 13:01:13.974164 27087 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:13 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:40464
I20260928 13:01:13.974195 26819 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:13 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:47746
I20260928 13:01:13.974208 26953 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:13 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:41776
I20260928 13:01:13.974434 27193 tablet_replica.cc:330] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: stopping tablet replica
I20260928 13:01:13.974506 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26756
I20260928 13:01:13.974538 27193 raft_consensus.cc:2229] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:01:13.974817 27193 raft_consensus.cc:2258] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:13.975258 27195 tablet_replica.cc:330] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: stopping tablet replica
I20260928 13:01:13.975435 27195 raft_consensus.cc:2229] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:01:13.975591 27195 raft_consensus.cc:2258] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:13.976169 27195 ts_tablet_manager.cc:1905] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:01:13.977909 27195 ts_tablet_manager.cc:1918] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:01:13.978003 27195 log.cc:1196] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/02a3d20c106a4ce1969b963c0cefe865
I20260928 13:01:13.978319 27195 ts_tablet_manager.cc:1939] T 02a3d20c106a4ce1969b963c0cefe865 P 39664385a0f1451a90a11ab2076ef5ef: Deleting consensus metadata
I20260928 13:01:13.978811 26679 catalog_manager.cc:4841] TS 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): tablet 02a3d20c106a4ce1969b963c0cefe865 (table test-table [id=7186f29cf08a403f9380760dbb348d1d]) successfully deleted
W20260928 13:01:13.980253 26681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:38537 (error 108)
I20260928 13:01:13.980381 27194 tablet_replica.cc:330] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: stopping tablet replica
I20260928 13:01:13.980538 27194 raft_consensus.cc:2229] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:01:13.980671 27194 raft_consensus.cc:2258] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:13.980854 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:13.980700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38537 (local address 127.0.0.1:47760)
0928 13:01:13.980767 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:13.980812 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 13:01:13.980906 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26890
W20260928 13:01:13.980983 26681 catalog_manager.cc:4584] TS fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02a3d20c106a4ce1969b963c0cefe865: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38537: connect: Connection refused (error 111)
I20260928 13:01:13.981649 27194 ts_tablet_manager.cc:1905] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:01:13.983721 27194 ts_tablet_manager.cc:1918] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:01:13.983807 27194 log.cc:1196] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/02a3d20c106a4ce1969b963c0cefe865
I20260928 13:01:13.984107 27194 ts_tablet_manager.cc:1939] T 02a3d20c106a4ce1969b963c0cefe865 P 69ee8725ef1a4a5898165a56837c7b18: Deleting consensus metadata
I20260928 13:01:13.984563 26681 catalog_manager.cc:4841] TS 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): tablet 02a3d20c106a4ce1969b963c0cefe865 (table test-table [id=7186f29cf08a403f9380760dbb348d1d]) successfully deleted
I20260928 13:01:13.986330 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27024
I20260928 13:01:13.991264 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 26662
I20260928 13:01:33.997136 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--webserver_interface=127.11.11.126
--webserver_port=44829
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:34.080142 27197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:34.080379 27197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:34.080403 27197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:34.081664 27197 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:01:34.081723 27197 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:01:34.081737 27197 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:01:34.081749 27197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:01:34.081763 27197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:34.081776 27197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:34.081821 27197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:01:34.081854 27197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:01:34.083463 27197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:34.083775 27197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:34.084064 27197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:34.093201 27203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:34.093425 27197 server_base.cc:1034] running on GCE node
W20260928 13:01:34.093307 27206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.093201 27204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:34.093840 27197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:34.094192 27197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:34.095371 27197 hybrid_clock.cc:648] HybridClock initialized: now 1790600494095368 us; error 38 us; skew 500 ppm
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:34.104453 27197 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:01:34.105022 27197 webserver.cc:460] Webserver started at http://127.11.11.126:44829/ using document root <none> and password file <none>
I20260928 13:01:34.105276 27197 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:34.105368 27197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:34.106886 27197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:34.107793 27213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.108105 27197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:34.108240 27197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "475a2fc8197b4b3595b6e8b16ceddbbd"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "2c006f75265d3f6a4434ba23235ab428"
server_key_iv: "35b757a48427f12f475899d3bcc28e03"
server_key_version: "encryptionkey@0"
I20260928 13:01:34.108771 27197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:34.122776 27197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:34.125401 27197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:34.125579 27197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:34.130141 27197 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46085
I20260928 13:01:34.130182 27265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46085 every 8 connection(s)
I20260928 13:01:34.130615 27197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:01:34.132368 27266 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:01:34.132654 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27197
I20260928 13:01:34.132961 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:38537
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=32957
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:01:34.133466 27266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap starting.
I20260928 13:01:34.144835 27266 log.cc:824] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:34.147512 27266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:01:34.147886 27266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap complete.
I20260928 13:01:34.151234 27266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:34.151611 27266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Initialized, Role: FOLLOWER
I20260928 13:01:34.151842 27266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:34.151937 27266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:01:34.151994 27266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:01:34.152068 27266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:01:34.153650 27266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:34.153777 27266 leader_election.cc:302] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475a2fc8197b4b3595b6e8b16ceddbbd; no voters:
I20260928 13:01:34.153954 27266 leader_election.cc:288] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:01:34.154059 27270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:01:34.154217 27266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:01:34.154336 27270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 LEADER]: Becoming Leader. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Running, Role: LEADER
I20260928 13:01:34.154587 27270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:34.154974 27271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:34.155066 27271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:34.155638 27272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475a2fc8197b4b3595b6e8b16ceddbbd. Latest consensus state: current_term: 2 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:34.155858 27272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:34.156207 27287 catalog_manager.cc:1259] Loaded cluster ID: d44db42b9701494192edaa477a9215f7
I20260928 13:01:34.156258 27287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: loading cluster ID for follower catalog manager: success
I20260928 13:01:34.157630 27287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: acquiring CA information for follower catalog manager: success
I20260928 13:01:34.158155 27287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:01:34.158257 27280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:01:34.158787 27280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:01:34.158892 27280 catalog_manager.cc:1259] Loaded cluster ID: d44db42b9701494192edaa477a9215f7
I20260928 13:01:34.158934 27280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:01:34.159132 27280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:01:34.159225 27280 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Loaded TSK: 0
I20260928 13:01:34.173952 27280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Generated new TSK 1
I20260928 13:01:34.174098 27280 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:01:34.226869 27268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:34.227113 27268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:34.227155 27268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:34.228468 27268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:34.228542 27268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:34.228591 27268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:01:34.230093 27268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:38537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=32957
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:34.230355 27268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:34.230603 27268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:34.235824 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.235823 27295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.235834 27294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:34.236030 27268 server_base.cc:1034] running on GCE node
I20260928 13:01:34.236647 27268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:34.236953 27268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:34.238164 27268 hybrid_clock.cc:648] HybridClock initialized: now 1790600494238137 us; error 49 us; skew 500 ppm
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:34.241227 27268 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:01:34.241839 27268 webserver.cc:460] Webserver started at http://127.11.11.65:32957/ using document root <none> and password file <none>
I20260928 13:01:34.242094 27268 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:34.242192 27268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:34.243795 27268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:34.244742 27304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.244956 27268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:01:34.245071 27268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "40852c363525ce642efdf1aa63518273"
server_key_iv: "b2a8f230d244ee5081f2a59f3152052c"
server_key_version: "encryptionkey@0"
I20260928 13:01:34.245456 27268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:34.254086 27268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:34.256721 27268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:34.256901 27268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:34.257172 27268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:34.257653 27311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:01:34.259840 27268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:01:34.259934 27268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:01:34.259992 27268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:01:34.260800 27268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:01:34.260861 27268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:01:34.260931 27311 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap starting.
I20260928 13:01:34.268858 27268 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38537
I20260928 13:01:34.269356 27268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:01:34.272226 27418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38537 every 8 connection(s)
I20260928 13:01:34.274080 27311 log.cc:824] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:34.277474 27311 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:34.277868 27311 tablet_bootstrap.cc:492] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap complete.
I20260928 13:01:34.279240 27311 ts_tablet_manager.cc:1397] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20260928 13:01:34.279385 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27268
I20260928 13:01:34.279681 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:39765
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=39103
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:01:34.281109 27311 raft_consensus.cc:348] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.281579 27311 raft_consensus.cc:729] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcfbe72b22a49d4b6346f8f0828653c, State: Initialized, Role: FOLLOWER
I20260928 13:01:34.281807 27311 consensus_queue.cc:260] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.282192 27311 ts_tablet_manager.cc:1428] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:01:34.283921 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.270313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:33887 (local address 127.11.11.126:46085)
0928 13:01:34.271241 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.271249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.271281 (+ 32us) server_negotiation.cc:407] Connection header received
0928 13:01:34.271412 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.271417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.271461 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.271615 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:34.272548 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.273539 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.274983 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.275442 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.275579 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.278248 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:34.278258 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:34.278259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:34.278270 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:34.281294 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.281966 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.281970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.281971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.282114 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.282461 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.282462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.282463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.282830 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:34.282901 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.283435 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.283612 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":116,"threads_started":1}
I20260928 13:01:34.284600 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.270253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:33887)
0928 13:01:34.270512 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.270562 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.271063 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.271646 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.271651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.271984 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:34.272399 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.272407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.273691 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.273692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.274882 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.274886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.275566 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.275568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.275713 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.276367 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:34.276377 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:34.278096 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:34.281379 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.281387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.281389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.281801 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.282208 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.282209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.282209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.282372 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.282961 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:34.282968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:34.283150 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.284088 (+ 938us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.284304 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":138,"threads_started":1}
I20260928 13:01:34.285794 27419 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:34.285907 27419 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:34.286124 27419 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:34.287070 27230 ts_manager.cc:194] Registered new tserver with Master: fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537)
I20260928 13:01:34.288546 27230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:33887
I20260928 13:01:34.289907 27419 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:34.296116 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.287826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38537 (local address 127.0.0.1:47858)
0928 13:01:34.289269 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.289311 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.289395 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.289913 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.289916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.289919 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:34.290018 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.290020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.291615 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.291617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.294589 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.294593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.295696 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.295697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.295804 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.295821 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.295900 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.295958 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"thread_start_us":98,"threads_started":1}
I20260928 13:01:34.296152 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.288902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.11.11.65:38537)
0928 13:01:34.289191 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.289196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.289484 (+ 288us) server_negotiation.cc:407] Connection header received
0928 13:01:34.289527 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.289530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.289728 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.289819 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:34.290115 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.291499 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.294712 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.295578 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.295666 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.295729 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.295932 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.295988 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
I20260928 13:01:34.296932 27353 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a3d20c106a4ce1969b963c0cefe865 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:13 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:47858
I20260928 13:01:34.297653 27428 tablet_replica.cc:330] stopping tablet replica
I20260928 13:01:34.297785 27428 raft_consensus.cc:2229] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:01:34.297878 27428 raft_consensus.cc:2258] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:34.298292 27428 ts_tablet_manager.cc:1905] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:01:34.301385 27428 ts_tablet_manager.cc:1918] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:01:34.301489 27428 log.cc:1196] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal/wals/02a3d20c106a4ce1969b963c0cefe865
I20260928 13:01:34.301801 27428 ts_tablet_manager.cc:1939] T 02a3d20c106a4ce1969b963c0cefe865 P fdcfbe72b22a49d4b6346f8f0828653c: Deleting consensus metadata
I20260928 13:01:34.302531 27217 catalog_manager.cc:4841] TS fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537): tablet 02a3d20c106a4ce1969b963c0cefe865 (table test-table [id=7186f29cf08a403f9380760dbb348d1d]) successfully deleted
W20260928 13:01:34.379102 27424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:34.379415 27424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:34.379467 27424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:34.380975 27424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:34.381071 27424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:34.381124 27424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:01:34.382990 27424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:39765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=39103
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:34.383384 27424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:34.383693 27424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:34.389333 27434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.389341 27436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.389345 27433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:34.389708 27424 server_base.cc:1034] running on GCE node
I20260928 13:01:34.389930 27424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:34.390200 27424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:34.391422 27424 hybrid_clock.cc:648] HybridClock initialized: now 1790600494391406 us; error 32 us; skew 500 ppm
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:34.394567 27424 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:01:34.395088 27424 webserver.cc:460] Webserver started at http://127.11.11.66:39103/ using document root <none> and password file <none>
I20260928 13:01:34.395371 27424 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:34.395478 27424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:34.397192 27424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:34.397964 27443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.398175 27424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:34.398290 27424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "39664385a0f1451a90a11ab2076ef5ef"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "6ea0538de03f396a0e691b44eefc70e2"
server_key_iv: "5ba42c19cabd6bd9d8b3bb11c69ff523"
server_key_version: "encryptionkey@0"
I20260928 13:01:34.398707 27424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:34.416395 27424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:34.418936 27424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:34.419142 27424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:34.419468 27424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:34.419823 27424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:34.419881 27424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.419917 27424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:34.419962 27424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.426467 27424 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:39765
I20260928 13:01:34.426554 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:39765 every 8 connection(s)
I20260928 13:01:34.426942 27424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:01:34.427310 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27424
I20260928 13:01:34.427666 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:42743
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=33729
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:34.439298 27559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.427875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:51519)
0928 13:01:34.428094 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.428147 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.428979 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.429352 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.429356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.429971 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:34.430183 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.430188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.431340 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.431341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.432472 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.432476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.433154 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.433155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.433272 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.433941 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:34.433950 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:34.435422 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:34.437737 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.437745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.437747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.438001 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.438244 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.438245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.438245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.438357 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.438741 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:34.438746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:34.438878 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.438969 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.439111 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260928 13:01:34.439306 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.427921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:51519 (local address 127.11.11.126:46085)
0928 13:01:34.427981 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.427984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.428259 (+ 275us) server_negotiation.cc:407] Connection header received
0928 13:01:34.429143 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.429146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.429168 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.429262 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:34.430293 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.431220 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.432572 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.433040 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.433129 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.435543 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:34.435545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:34.435546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:34.435557 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:34.437623 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.438085 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.438087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.438087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.438163 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.438472 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.438474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.438474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.438658 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:34.438742 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.439030 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.439151 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 13:01:34.439992 27557 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:34.440140 27557 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:34.440358 27557 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:34.440833 27230 ts_manager.cc:194] Registered new tserver with Master: 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765)
I20260928 13:01:34.441428 27230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:51519
W20260928 13:01:34.517215 27560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:34.517436 27560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:34.517460 27560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:34.518579 27560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:34.518615 27560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:34.518642 27560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:01:34.520169 27560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:42743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=33729
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:34.520372 27560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:34.520577 27560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:34.525693 27566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.525768 27568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:34.525764 27565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:34.525831 27560 server_base.cc:1034] running on GCE node
I20260928 13:01:34.526150 27560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:34.526433 27560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:34.527632 27560 hybrid_clock.cc:648] HybridClock initialized: now 1790600494527619 us; error 48 us; skew 500 ppm
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:34.530218 27560 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:01:34.530747 27560 webserver.cc:460] Webserver started at http://127.11.11.67:33729/ using document root <none> and password file <none>
I20260928 13:01:34.530987 27560 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:34.531073 27560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:34.532570 27560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:34.533157 27575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.533346 27560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:01:34.533478 27560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "69ee8725ef1a4a5898165a56837c7b18"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "8cf9f1c1dc2781e111826f3a05901b14"
server_key_iv: "b6d8d885c3f1b45263ae0e4e4b6939e0"
server_key_version: "encryptionkey@0"
I20260928 13:01:34.533890 27560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:34.554304 27560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:34.556519 27560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:34.556691 27560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:34.556941 27560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:34.557277 27560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:34.557337 27560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.557392 27560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:34.557417 27560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:34.563495 27560 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:42743
I20260928 13:01:34.563539 27688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:42743 every 8 connection(s)
I20260928 13:01:34.563980 27560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:01:34.564163 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27560
Sep 28 13:01:34 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600494, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:01:34.571317 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.564601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.11.11.126:46085)
0928 13:01:34.564667 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.564670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.564868 (+ 198us) server_negotiation.cc:407] Connection header received
0928 13:01:34.564959 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.564960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.564971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.565043 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:34.565648 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.566152 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.566953 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.567163 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.567241 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.568587 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:34.568588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:34.568589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:34.568596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:34.570124 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.570483 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.570485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.570486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.570546 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.570782 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.570783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.570783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.570904 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:34.571100 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.571136 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.571213 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 13:01:34.573596 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.564736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:59795 (local address 127.11.11.126:46085)
0928 13:01:34.564854 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.564856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.565066 (+ 210us) server_negotiation.cc:407] Connection header received
0928 13:01:34.565619 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.565620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.565632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.565696 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:34.566633 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.567131 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.568180 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.568395 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.568478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.570961 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:34.570962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:34.570963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:34.570970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:34.572512 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.572861 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.572862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.572862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.572913 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.573087 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.573087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.573088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.573180 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:34.573230 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.573470 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.573532 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260928 13:01:34.573767 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.564776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:59795)
0928 13:01:34.564941 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.564994 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.565514 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.565772 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.565777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.566337 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:34.566538 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.566542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.567231 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.567232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.568077 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.568081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.568490 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.568491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.568624 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.569251 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:34.569260 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:34.570869 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:34.572581 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.572586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.572588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.572798 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.572953 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:34.572953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:34.572954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:34.573038 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:34.573236 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:34.573240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:34.573354 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.573442 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.573561 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 13:01:34.574235 27689 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46085
I20260928 13:01:34.574354 27689 heartbeater.cc:464] Registering TS with master...
I20260928 13:01:34.574545 27689 heartbeater.cc:511] Master 127.11.11.126:46085 requested a full tablet report, sending...
I20260928 13:01:34.574896 27230 ts_manager.cc:194] Registered new tserver with Master: 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
I20260928 13:01:34.575259 27230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:59795
I20260928 13:01:34.583122 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:01:34.585862 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.583542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.11.11.126:46085)
0928 13:01:34.583614 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.583616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.583690 (+ 74us) server_negotiation.cc:407] Connection header received
0928 13:01:34.583747 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.583748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.583784 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.583850 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:01:34.583952 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.584483 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.585241 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.585439 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.585496 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.585646 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:01:34.585693 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 13:01:34.585810 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49056: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 13:01:34.585950 27692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49056: token expired
W20260928 13:01:34.586054 27158 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.11.126:46085)
I20260928 13:01:34.591840 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.586322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.11.11.126:46085)
0928 13:01:34.586355 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.586356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.586364 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:01:34.586425 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.586425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.586432 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.586472 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:34.586935 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.587357 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.588010 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.588243 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.588294 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.589417 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:34.589418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:34.589418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:34.589424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:34.590912 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.591220 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.591221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.591221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.591269 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:34.591442 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:34.591443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:34.591443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:34.591563 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:34.591635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.591720 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.591777 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 13:01:34.592993 27230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:01:34.594624 27230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:01:34.597203 27353 tablet_service.cc:1467] Processing CreateTablet for tablet 2b07e2449836466faea7416421ead4cf (DEFAULT_TABLE table=test-table [id=54cc5f5156ad465499a34976eca670b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:34.597425 27353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b07e2449836466faea7416421ead4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:34.599064 27311 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap starting.
I20260928 13:01:34.599443 27311 tablet_bootstrap.cc:654] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:34.600400 27311 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: No bootstrap required, opened a new log
I20260928 13:01:34.600481 27311 ts_tablet_manager.cc:1397] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:01:34.600610 27311 raft_consensus.cc:348] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.600696 27311 raft_consensus.cc:374] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:34.600760 27311 raft_consensus.cc:729] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcfbe72b22a49d4b6346f8f0828653c, State: Initialized, Role: FOLLOWER
I20260928 13:01:34.600822 27311 consensus_queue.cc:260] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.600955 27311 ts_tablet_manager.cc:1428] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:01:34.601089 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.596668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39765 (local address 127.0.0.1:52300)
0928 13:01:34.596722 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.596735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.596786 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.597315 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.597316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.597318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:34.597377 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.597379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.598706 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.598707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.599917 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.599921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.600830 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.600831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.600910 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.600914 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.600965 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.601013 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260928 13:01:34.601445 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.596916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.11.11.66:39765)
0928 13:01:34.597123 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.597128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.597139 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:01:34.597182 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.597184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.597200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.597457 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:34.597554 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.598610 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.599997 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.600713 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.601191 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.601271 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.601303 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.601348 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260928 13:01:34.602103 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.596694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:42743 (local address 127.0.0.1:45074)
0928 13:01:34.596853 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.596863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.596911 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.597592 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.597593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.597596 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:34.597697 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.597700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.598818 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.598819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.599849 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.599851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.601795 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.601795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.601882 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.601886 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.601973 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.602022 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":113,"thread_start_us":110,"threads_started":1}
I20260928 13:01:34.602113 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.597131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.11.11.67:42743)
0928 13:01:34.597300 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.597345 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.597357 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:01:34.597404 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.597407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.597426 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.597795 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:34.597905 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.598726 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.599940 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.601708 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.601796 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.601855 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.601987 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.602031 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260928 13:01:34.602264 27491 tablet_service.cc:1467] Processing CreateTablet for tablet 2b07e2449836466faea7416421ead4cf (DEFAULT_TABLE table=test-table [id=54cc5f5156ad465499a34976eca670b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:34.602558 27491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b07e2449836466faea7416421ead4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:34.604681 27623 tablet_service.cc:1467] Processing CreateTablet for tablet 2b07e2449836466faea7416421ead4cf (DEFAULT_TABLE table=test-table [id=54cc5f5156ad465499a34976eca670b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:01:34.605090 27623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b07e2449836466faea7416421ead4cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:01:34.605829 27699 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Bootstrap starting.
I20260928 13:01:34.606459 27699 tablet_bootstrap.cc:654] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:34.606932 27699 log.cc:824] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:34.607542 27701 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Bootstrap starting.
I20260928 13:01:34.607694 27699 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: No bootstrap required, opened a new log
I20260928 13:01:34.607775 27699 ts_tablet_manager.cc:1397] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:01:34.608207 27701 tablet_bootstrap.cc:654] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Neither blocks nor log segments found. Creating new log.
I20260928 13:01:34.608798 27701 log.cc:824] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:34.609465 27701 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: No bootstrap required, opened a new log
I20260928 13:01:34.609517 27701 ts_tablet_manager.cc:1397] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:01:34.609527 27699 raft_consensus.cc:348] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.609731 27699 raft_consensus.cc:374] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:34.609782 27699 raft_consensus.cc:729] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39664385a0f1451a90a11ab2076ef5ef, State: Initialized, Role: FOLLOWER
I20260928 13:01:34.609915 27699 consensus_queue.cc:260] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.610222 27699 ts_tablet_manager.cc:1428] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:01:34.610247 27557 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:34.611515 27701 raft_consensus.cc:348] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.611688 27701 raft_consensus.cc:374] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:01:34.611752 27701 raft_consensus.cc:729] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ee8725ef1a4a5898165a56837c7b18, State: Initialized, Role: FOLLOWER
I20260928 13:01:34.611863 27701 consensus_queue.cc:260] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.612196 27701 ts_tablet_manager.cc:1428] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:01:34.612207 27689 heartbeater.cc:503] Master 127.11.11.126:46085 was elected leader, sending a full tablet report...
I20260928 13:01:34.644357 27704 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:34.644479 27704 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.644786 27704 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537)
I20260928 13:01:34.647754 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.644966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39765 (local address 127.11.11.67:42213)
0928 13:01:34.645049 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.645061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.645105 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.645293 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.645294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.645296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:34.645378 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.645380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.646111 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.646112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.646876 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.646878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.647469 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.647470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.647557 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.647560 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.647629 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.647668 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":18}
I20260928 13:01:34.647744 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.645075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:42213 (local address 127.11.11.66:39765)
0928 13:01:34.645135 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.645137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.645149 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:01:34.645180 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.645181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.645190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.645288 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:34.645486 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.646023 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.646975 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.647364 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.647469 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.647478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.647633 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.647669 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:01:34.647744 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.645152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:55605 (local address 127.11.11.65:38537)
0928 13:01:34.645213 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.645215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.645251 (+ 36us) server_negotiation.cc:407] Connection header received
0928 13:01:34.645329 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.645331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.645339 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.645392 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:01:34.645580 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.646045 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.646982 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.647356 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.647451 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.647460 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.647618 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.647665 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 13:01:34.647776 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.644966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38537 (local address 127.11.11.67:55605)
0928 13:01:34.645204 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.645213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:34.645262 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:34.645438 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:34.645440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:34.645441 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:01:34.645509 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.645510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.646128 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.646128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.646890 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:34.646892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.647453 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.647454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:34.647529 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.647533 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:01:34.647635 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 13:01:34.647674 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":45,"thread_start_us":119,"threads_started":1}
I20260928 13:01:34.648129 27373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b07e2449836466faea7416421ead4cf" candidate_uuid: "69ee8725ef1a4a5898165a56837c7b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" is_pre_election: true
I20260928 13:01:34.648202 27511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b07e2449836466faea7416421ead4cf" candidate_uuid: "69ee8725ef1a4a5898165a56837c7b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39664385a0f1451a90a11ab2076ef5ef" is_pre_election: true
I20260928 13:01:34.648288 27373 raft_consensus.cc:2454] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ee8725ef1a4a5898165a56837c7b18 in term 0.
I20260928 13:01:34.648368 27511 raft_consensus.cc:2454] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ee8725ef1a4a5898165a56837c7b18 in term 0.
I20260928 13:01:34.648514 27579 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ee8725ef1a4a5898165a56837c7b18, fdcfbe72b22a49d4b6346f8f0828653c; no voters:
I20260928 13:01:34.648661 27704 raft_consensus.cc:2781] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:01:34.648739 27704 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:01:34.648779 27704 raft_consensus.cc:3037] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:34.649549 27704 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.649696 27704 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [CANDIDATE]: Term 1 election: Requested vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537)
I20260928 13:01:34.649873 27373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b07e2449836466faea7416421ead4cf" candidate_uuid: "69ee8725ef1a4a5898165a56837c7b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
I20260928 13:01:34.649974 27373 raft_consensus.cc:3037] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:34.651003 27373 raft_consensus.cc:2454] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ee8725ef1a4a5898165a56837c7b18 in term 1.
I20260928 13:01:34.651161 27511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b07e2449836466faea7416421ead4cf" candidate_uuid: "69ee8725ef1a4a5898165a56837c7b18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39664385a0f1451a90a11ab2076ef5ef"
I20260928 13:01:34.651203 27579 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ee8725ef1a4a5898165a56837c7b18, fdcfbe72b22a49d4b6346f8f0828653c; no voters:
I20260928 13:01:34.651285 27704 raft_consensus.cc:2781] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:01:34.651262 27511 raft_consensus.cc:3037] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:01:34.651465 27704 raft_consensus.cc:686] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 1 LEADER]: Becoming Leader. State: Replica: 69ee8725ef1a4a5898165a56837c7b18, State: Running, Role: LEADER
I20260928 13:01:34.651573 27704 consensus_queue.cc:237] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:34.652235 27511 raft_consensus.cc:2454] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ee8725ef1a4a5898165a56837c7b18 in term 1.
I20260928 13:01:34.652765 27229 catalog_manager.cc:5495] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "69ee8725ef1a4a5898165a56837c7b18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } health_report { overall_health: UNKNOWN } } }
I20260928 13:01:34.659039 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.656932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.11.11.67:42743)
0928 13:01:34.656988 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:34.656990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:34.656999 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:01:34.657067 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:34.657068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:34.657083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:34.657133 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:01:34.657226 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.657671 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.658372 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:34.658548 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:34.658595 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:34.658754 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:01:34.658846 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:01:34.658919 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:34.658936 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 13:01:34.658952 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:01:34.660938 27373 raft_consensus.cc:1261] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Refusing update from remote peer 69ee8725ef1a4a5898165a56837c7b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:01:34.660938 27511 raft_consensus.cc:1261] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Refusing update from remote peer 69ee8725ef1a4a5898165a56837c7b18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:01:34.661319 27704 consensus_queue.cc:1035] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:01:34.661437 27706 consensus_queue.cc:1035] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:01:34.664140 27710 mvcc.cc:205] Tried to move back new op lower bound from 7334299626128322560 to 7334299626093654016. Current Snapshot: MvccSnapshot[applied={T|T < 7334299626128322560}]
I20260928 13:01:34.664244 27711 mvcc.cc:205] Tried to move back new op lower bound from 7334299626128322560 to 7334299626093654016. Current Snapshot: MvccSnapshot[applied={T|T < 7334299626128322560}]
I20260928 13:01:34.666054 27229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:01:34.666155 27229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:01:34.666968 27229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Sending DeleteTablet for 3 replicas of tablet 2b07e2449836466faea7416421ead4cf
I20260928 13:01:34.667313 27353 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b07e2449836466faea7416421ead4cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:34 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:47858
I20260928 13:01:34.667300 27491 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b07e2449836466faea7416421ead4cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:34 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:52300
I20260928 13:01:34.667420 27428 tablet_replica.cc:330] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: stopping tablet replica
I20260928 13:01:34.667462 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27268
I20260928 13:01:34.667492 27428 raft_consensus.cc:2229] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:01:34.667552 27428 raft_consensus.cc:2258] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:34.668092 27718 tablet_replica.cc:330] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: stopping tablet replica
I20260928 13:01:34.668207 27718 raft_consensus.cc:2229] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:01:34.668344 27718 raft_consensus.cc:2258] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:34.668744 27718 ts_tablet_manager.cc:1905] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:01:34.670598 27718 ts_tablet_manager.cc:1918] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:01:34.670691 27718 log.cc:1196] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/2b07e2449836466faea7416421ead4cf
I20260928 13:01:34.670830 27623 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b07e2449836466faea7416421ead4cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:01:34 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:45074
I20260928 13:01:34.671015 27718 ts_tablet_manager.cc:1939] T 2b07e2449836466faea7416421ead4cf P 39664385a0f1451a90a11ab2076ef5ef: Deleting consensus metadata
I20260928 13:01:34.671571 27215 catalog_manager.cc:4841] TS 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): tablet 2b07e2449836466faea7416421ead4cf (table test-table [id=54cc5f5156ad465499a34976eca670b5]) successfully deleted
W20260928 13:01:34.675449 27217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:38537 (error 108)
I20260928 13:01:34.675808 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27424
I20260928 13:01:34.675966 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:34.675820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38537 (local address 127.0.0.1:47868)
0928 13:01:34.675894 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:34.675929 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:01:34.676507 27719 tablet_replica.cc:330] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: stopping tablet replica
I20260928 13:01:34.676631 27719 raft_consensus.cc:2229] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:01:34.676743 27719 raft_consensus.cc:2258] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:01:34.677999 27719 ts_tablet_manager.cc:1905] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 13:01:34.677727 27217 catalog_manager.cc:4584] TS fdcfbe72b22a49d4b6346f8f0828653c (127.11.11.65:38537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b07e2449836466faea7416421ead4cf: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38537: connect: Connection refused (error 111)
I20260928 13:01:34.679960 27719 ts_tablet_manager.cc:1918] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:01:34.680109 27719 log.cc:1196] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/2b07e2449836466faea7416421ead4cf
I20260928 13:01:34.680459 27719 ts_tablet_manager.cc:1939] T 2b07e2449836466faea7416421ead4cf P 69ee8725ef1a4a5898165a56837c7b18: Deleting consensus metadata
I20260928 13:01:34.680946 27217 catalog_manager.cc:4841] TS 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): tablet 2b07e2449836466faea7416421ead4cf (table test-table [id=54cc5f5156ad465499a34976eca670b5]) successfully deleted
I20260928 13:01:34.681849 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27560
I20260928 13:01:34.687224 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27197
I20260928 13:01:54.693216 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--webserver_interface=127.11.11.126
--webserver_port=44829
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:01:54.776453 27721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:54.776736 27721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:54.776775 27721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:54.778007 27721 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:01:54.778084 27721 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:01:54.778105 27721 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:01:54.778124 27721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:01:54.778152 27721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:54.778182 27721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:54.778208 27721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:01:54.778237 27721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:01:54.779784 27721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:54.780123 27721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:54.780366 27721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:01:54.785567 27721 server_base.cc:1034] running on GCE node
W20260928 13:01:54.785583 27726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:54.785573 27729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:54.785504 27727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:54.785975 27721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:54.786266 27721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:54.787417 27721 hybrid_clock.cc:648] HybridClock initialized: now 1790600514787402 us; error 26 us; skew 500 ppm
Sep 28 13:01:54 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600514, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:54.790232 27721 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:01:54.790740 27721 webserver.cc:460] Webserver started at http://127.11.11.126:44829/ using document root <none> and password file <none>
I20260928 13:01:54.790935 27721 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:54.790978 27721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:54.792547 27721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:01:54.793344 27736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:54.793614 27721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:01:54.793702 27721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "475a2fc8197b4b3595b6e8b16ceddbbd"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "2c006f75265d3f6a4434ba23235ab428"
server_key_iv: "35b757a48427f12f475899d3bcc28e03"
server_key_version: "encryptionkey@0"
I20260928 13:01:54.794101 27721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:54.801342 27721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:54.803586 27721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:54.803747 27721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:54.807816 27721 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46085
I20260928 13:01:54.807885 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46085 every 8 connection(s)
I20260928 13:01:54.808308 27721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:01:54.808518 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27721
I20260928 13:01:54.808826 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:38537
--local_ip_for_outbound_sockets=127.11.11.65
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=32957
--webserver_interface=127.11.11.65
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:01:54.809911 27789 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:01:54.810622 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap starting.
I20260928 13:01:54.819427 27789 log.cc:824] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:54.821913 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:01:54.822311 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Bootstrap complete.
I20260928 13:01:54.824983 27789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:54.825227 27789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Initialized, Role: FOLLOWER
I20260928 13:01:54.825399 27789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:54.825470 27789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:01:54.825531 27789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:01:54.825578 27789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 2 FOLLOWER]: Advancing to term 3
I20260928 13:01:54.826639 27789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:54.826768 27789 leader_election.cc:302] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 475a2fc8197b4b3595b6e8b16ceddbbd; no voters:
I20260928 13:01:54.827008 27789 leader_election.cc:288] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 13:01:54.827047 27793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 13:01:54.827239 27793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [term 3 LEADER]: Becoming Leader. State: Replica: 475a2fc8197b4b3595b6e8b16ceddbbd, State: Running, Role: LEADER
I20260928 13:01:54.827301 27789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:01:54.827361 27793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } }
I20260928 13:01:54.827740 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:54.827934 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:54.827737 27793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475a2fc8197b4b3595b6e8b16ceddbbd. Latest consensus state: current_term: 3 leader_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "475a2fc8197b4b3595b6e8b16ceddbbd" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46085 } } }
I20260928 13:01:54.828397 27802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:01:54.828487 27793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd [sys.catalog]: This master's current role is: LEADER
I20260928 13:01:54.829445 27802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:01:54.829567 27802 catalog_manager.cc:1259] Loaded cluster ID: d44db42b9701494192edaa477a9215f7
I20260928 13:01:54.829622 27802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:01:54.830879 27802 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:01:54.831073 27802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Loaded TSK: 0
I20260928 13:01:54.831120 27802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 475a2fc8197b4b3595b6e8b16ceddbbd: Loaded TSK: 1
F20260928 13:01:54.831354 27802 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f436d3029ad google::LogMessage::Fail() at ??:0
@ 0x7f436d30588a google::LogMessage::SendToLog() at ??:0
@ 0x7f436d3024de google::LogMessage::Flush() at ??:0
@ 0x7f436d303a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f436fa6f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f436fa6cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f437525c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f437525c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f437525c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4375222c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f437522cad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f436f1d00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f436f1c5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43700296db start_thread at ??:0
@ 0x7f436be0061f clone at ??:0
W20260928 13:01:54.932528 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:54.932835 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:54.932874 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:54.934607 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:54.934684 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:54.934727 27791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:01:54.937040 27791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:38537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=32957
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:54.937332 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:54.937621 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:54.944340 27833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:54.944656 27830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:54.944342 27831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:54.944865 27791 server_base.cc:1034] running on GCE node
I20260928 13:01:54.945293 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:54.945574 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:54.946760 27791 hybrid_clock.cc:648] HybridClock initialized: now 1790600514946732 us; error 45 us; skew 500 ppm
Sep 28 13:01:54 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600514, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:54.950088 27791 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:01:54.950639 27791 webserver.cc:460] Webserver started at http://127.11.11.65:32957/ using document root <none> and password file <none>
I20260928 13:01:54.950929 27791 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:54.951011 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:54.953023 27791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:54.954514 27840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:54.955237 27791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:54.955343 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "fdcfbe72b22a49d4b6346f8f0828653c"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "40852c363525ce642efdf1aa63518273"
server_key_iv: "b2a8f230d244ee5081f2a59f3152052c"
server_key_version: "encryptionkey@0"
I20260928 13:01:54.955826 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:54.977252 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:54.980137 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:54.980304 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:54.980592 27791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:54.981186 27847 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:01:54.983542 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:01:54.983632 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 13:01:54.983677 27791 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:01:54.984645 27791 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:01:54.984692 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:01:54.984875 27847 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap starting.
I20260928 13:01:54.992362 27791 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38537
I20260928 13:01:54.992815 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:01:54.993412 27954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38537 every 8 connection(s)
I20260928 13:01:54.998112 27847 log.cc:824] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:01:54.999981 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27791
I20260928 13:01:55.000383 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:39765
--local_ip_for_outbound_sockets=127.11.11.66
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=39103
--webserver_interface=127.11.11.66
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:01:55.001469 27847 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:01:55.001784 27847 tablet_bootstrap.cc:492] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Bootstrap complete.
I20260928 13:01:55.003055 27847 ts_tablet_manager.cc:1397] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20260928 13:01:55.004232 27847 raft_consensus.cc:348] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:55.004556 27847 raft_consensus.cc:729] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcfbe72b22a49d4b6346f8f0828653c, State: Initialized, Role: FOLLOWER
I20260928 13:01:55.004719 27847 consensus_queue.cc:260] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:55.005015 27847 ts_tablet_manager.cc:1428] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 13:01:55.091861 27957 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:01:54.993552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:56955)
0928 13:01:54.993767 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:54.993816 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:54.994435 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:55.091620 (+ 97185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
W20260928 13:01:55.092110 27955 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:46085 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:46085: Client connection negotiation failed: client connection to 127.11.11.126:46085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:01:55.092480 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.092349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:49995)
0928 13:01:55.092405 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.092460 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:01:55.092927 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.092837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:40777)
0928 13:01:55.092887 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.092911 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 13:01:55.093057 27955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:01:55.112609 27959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:55.112833 27959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:55.112859 27959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:55.114071 27959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:55.114118 27959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:55.114174 27959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:01:55.115674 27959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:39765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=39103
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:55.115926 27959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:55.116201 27959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:55.121186 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:55.121200 27965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:55.121323 27966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:55.121591 27959 server_base.cc:1034] running on GCE node
I20260928 13:01:55.121816 27959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:55.122063 27959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:55.123227 27959 hybrid_clock.cc:648] HybridClock initialized: now 1790600515123218 us; error 42 us; skew 500 ppm
Sep 28 13:01:55 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600515, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:55.137841 27959 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:01:55.138290 27959 webserver.cc:460] Webserver started at http://127.11.11.66:39103/ using document root <none> and password file <none>
I20260928 13:01:55.138489 27959 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:55.138532 27959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:55.139895 27959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:55.140664 27975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.140880 27959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:01:55.140991 27959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "39664385a0f1451a90a11ab2076ef5ef"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "6ea0538de03f396a0e691b44eefc70e2"
server_key_iv: "5ba42c19cabd6bd9d8b3bb11c69ff523"
server_key_version: "encryptionkey@0"
I20260928 13:01:55.141368 27959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:55.154716 27959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:55.157421 27959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:55.157560 27959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:55.157768 27959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:55.158123 27959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:55.158166 27959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.158216 27959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:55.158246 27959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.165055 27959 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:39765
I20260928 13:01:55.165120 28088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:39765 every 8 connection(s)
I20260928 13:01:55.165534 27959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:01:55.166671 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.166285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:45125)
0928 13:01:55.166441 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.166516 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260928 13:01:55.166922 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 27959
W20260928 13:01:55.167179 28089 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:46085 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:46085: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
I20260928 13:01:55.167218 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:42743
--local_ip_for_outbound_sockets=127.11.11.67
--tserver_master_addrs=127.11.11.126:46085
--webserver_port=33729
--webserver_interface=127.11.11.67
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:44005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:01:55.167503 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.167396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:58393)
0928 13:01:55.167437 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.167470 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 13:01:55.167922 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.167815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:57379)
0928 13:01:55.167866 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.167894 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:01:55.168022 28089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:01:55.248678 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:01:55.248901 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:01:55.248927 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:01:55.250064 28092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:01:55.250100 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:01:55.250126 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:01:55.251551 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:44005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:42743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=33729
--tserver_master_addrs=127.11.11.126:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:01:55.251787 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:01:55.252010 28092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:01:55.256986 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:55.257092 28092 server_base.cc:1034] running on GCE node
W20260928 13:01:55.257208 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:01:55.257345 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:01:55.257575 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:01:55.257787 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:01:55.258944 28092 hybrid_clock.cc:648] HybridClock initialized: now 1790600515258931 us; error 27 us; skew 500 ppm
Sep 28 13:01:55 dist-test-slave-8sjz krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600515, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:01:55.261824 28092 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:01:55.262290 28092 webserver.cc:460] Webserver started at http://127.11.11.67:33729/ using document root <none> and password file <none>
I20260928 13:01:55.262534 28092 fs_manager.cc:359] Metadata directory not provided
I20260928 13:01:55.262581 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:01:55.264082 28092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:01:55.264752 28107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.264982 28092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:01:55.265074 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "69ee8725ef1a4a5898165a56837c7b18"
format_stamp: "Formatted at 2026-09-28 13:01:13 on dist-test-slave-8sjz"
server_key: "8cf9f1c1dc2781e111826f3a05901b14"
server_key_iv: "b6d8d885c3f1b45263ae0e4e4b6939e0"
server_key_version: "encryptionkey@0"
I20260928 13:01:55.265451 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:01:55.288192 28092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:01:55.290776 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:01:55.290963 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:01:55.291242 28092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:01:55.291605 28092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:01:55.291666 28092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.291713 28092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:01:55.291742 28092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:01:55.298200 28092 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:42743
I20260928 13:01:55.298254 28220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:42743 every 8 connection(s)
I20260928 13:01:55.298635 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:01:55.299715 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.299333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:42437)
0928 13:01:55.299487 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.299560 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260928 13:01:55.300171 28221 heartbeater.cc:644] Failed to heartbeat to 127.11.11.126:46085 (0 consecutive failures): Network error: Failed to ping master at 127.11.11.126:46085: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
I20260928 13:01:55.300480 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.300387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:39875)
0928 13:01:55.300435 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.300463 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 13:01:55.300808 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:55.300757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:55959)
0928 13:01:55.300779 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:55.300797 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 13:01:55.300937 28221 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:01:55.302680 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28092
I20260928 13:01:56.093766 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.093516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:56469)
0928 13:01:56.093653 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:56.093699 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260928 13:01:56.168672 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.168433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:60997)
0928 13:01:56.168581 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:56.168619 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260928 13:01:56.301533 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.301313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:37757)
0928 13:01:56.301451 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:56.301483 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 13:01:56.324681 28228 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:56.324841 28228 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:56.325166 28228 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
Sep 28 13:01:56 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600514, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.66@KRBTEST.COM
Sep 28 13:01:56 dist-test-slave-8sjz krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600514, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.67@KRBTEST.COM
I20260928 13:01:56.335600 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.325349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:39765 (local address 127.11.11.65:45047)
0928 13:01:56.325469 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:56.325479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:56.325519 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:56.326373 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:56.326374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:56.326743 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:56.326947 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:56.326953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.328025 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.328026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:56.328841 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:56.328843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.329456 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.329457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:56.329563 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:56.330201 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:56.330205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:56.331723 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:56.334037 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:56.334043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:56.334045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:56.334261 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:56.334551 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:56.334551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:56.334552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:56.334665 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:56.335069 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:56.335076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:56.335212 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 13:01:56.335322 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 13:01:56.335490 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":7,"thread_start_us":64,"threads_started":1}
I20260928 13:01:56.335613 28231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.325467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:45047 (local address 127.11.11.66:39765)
0928 13:01:56.325630 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:56.325641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:56.325669 (+ 28us) server_negotiation.cc:407] Connection header received
0928 13:01:56.326123 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:56.326125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:56.326162 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:56.326317 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:56.327091 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.327932 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.328930 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.329341 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.329474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:56.331829 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:56.331832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:56.331832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:56.331843 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:56.333962 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:56.334370 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:56.334373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:56.334373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:56.334475 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:56.334740 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:56.334740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:56.334741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:56.334990 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:56.335077 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:56.335380 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 13:01:56.335506 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260928 13:01:56.336072 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.325336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:42743 (local address 127.11.11.65:49205)
0928 13:01:56.325403 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:56.325414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:01:56.325483 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:01:56.326359 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:01:56.326363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:01:56.326742 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:01:56.326948 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:56.326953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.328126 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.328127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:56.328842 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:01:56.328844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.329447 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.329449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:01:56.329578 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:56.330145 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:01:56.330153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:01:56.332211 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:01:56.334808 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:56.334808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:56.334809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:56.334972 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:56.335255 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:01:56.335257 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:01:56.335257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:01:56.335320 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:01:56.335747 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:01:56.335748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:01:56.335805 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 13:01:56.335861 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 13:01:56.335957 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":65}
W20260928 13:01:56.336431 27842 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
I20260928 13:01:56.336532 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:56.325461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:49205 (local address 127.11.11.67:42743)
0928 13:01:56.325632 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 13:01:56.325641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:01:56.325675 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:01:56.326105 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:01:56.326108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:01:56.326149 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:01:56.326273 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:01:56.327081 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.328008 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.328949 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:01:56.329344 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:01:56.329463 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:01:56.332318 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:01:56.332321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:01:56.332321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:01:56.332331 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:01:56.334706 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:56.335081 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:56.335084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:56.335084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:56.335185 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:01:56.335396 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:01:56.335396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:01:56.335397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:01:56.335633 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:01:56.336190 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0928 13:01:56.336265 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 13:01:56.336398 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
W20260928 13:01:56.337185 27844 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
I20260928 13:01:56.337236 27844 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdcfbe72b22a49d4b6346f8f0828653c; no voters: 39664385a0f1451a90a11ab2076ef5ef, 69ee8725ef1a4a5898165a56837c7b18
I20260928 13:01:56.337358 28228 raft_consensus.cc:2726] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:57.094645 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:57.094368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:35095)
0928 13:01:57.094527 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:57.094584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260928 13:01:57.169329 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:57.169099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:58119)
0928 13:01:57.169248 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:57.169286 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260928 13:01:57.302214 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:57.301990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:59095)
0928 13:01:57.302132 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:57.302173 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260928 13:01:58.095408 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:58.095140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:36451)
0928 13:01:58.095310 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:58.095358 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260928 13:01:58.170079 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:58.169872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:33525)
0928 13:01:58.169999 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:58.170039 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260928 13:01:58.303010 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:58.302681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:51939)
0928 13:01:58.302907 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:58.302956 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260928 13:01:58.568964 28238 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:01:58.569087 28238 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:01:58.569319 28238 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
W20260928 13:01:58.569754 27842 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
W20260928 13:01:58.569824 27844 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
I20260928 13:01:58.569904 27842 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdcfbe72b22a49d4b6346f8f0828653c; no voters: 39664385a0f1451a90a11ab2076ef5ef, 69ee8725ef1a4a5898165a56837c7b18
I20260928 13:01:58.569999 28238 raft_consensus.cc:2726] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:01:59.096326 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:59.096027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:36227)
0928 13:01:59.096231 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:59.096276 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260928 13:01:59.170746 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:59.170527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:56093)
0928 13:01:59.170652 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:59.170699 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 13:01:59.303912 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:01:59.303689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:59471)
0928 13:01:59.303833 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:01:59.303873 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 13:02:00.097220 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:00.096895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:56083)
0928 13:02:00.097082 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:00.097132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260928 13:02:00.171535 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:00.171285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:56411)
0928 13:02:00.171440 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:00.171472 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260928 13:02:00.304649 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:00.304404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:55329)
0928 13:02:00.304538 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:00.304584 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 13:02:00.365844 28245 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:00.365950 28245 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:02:00.366117 28245 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
W20260928 13:02:00.366509 27842 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
W20260928 13:02:00.366601 27844 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
I20260928 13:02:00.366698 27844 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdcfbe72b22a49d4b6346f8f0828653c; no voters: 39664385a0f1451a90a11ab2076ef5ef, 69ee8725ef1a4a5898165a56837c7b18
I20260928 13:02:00.366799 28245 raft_consensus.cc:2726] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:02:01.098120 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:01.097875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:40173)
0928 13:02:01.098010 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:01.098052 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260928 13:02:01.172348 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:01.172084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:51363)
0928 13:02:01.172248 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:01.172299 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260928 13:02:01.305384 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:01.305123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:60969)
0928 13:02:01.305288 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:01.305336 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 13:02:02.098904 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:02.098685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:33183)
0928 13:02:02.098818 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:02.098856 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 13:02:02.173116 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:02.172878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:48683)
0928 13:02:02.173024 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:02.173063 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 13:02:02.306209 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:02.305973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:50203)
0928 13:02:02.306116 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:02.306162 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 13:02:03.099705 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:03.099462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:45449)
0928 13:02:03.099608 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:03.099653 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260928 13:02:03.173995 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:03.173747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:55275)
0928 13:02:03.173899 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:03.173946 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 13:02:03.307003 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:03.306763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:51173)
0928 13:02:03.306902 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:03.306947 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 13:02:04.100582 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:04.100297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:38859)
0928 13:02:04.100443 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:04.100523 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260928 13:02:04.174798 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:04.174495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:35733)
0928 13:02:04.174710 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:04.174750 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":140,"threads_started":1}
I20260928 13:02:04.307891 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:04.307645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:49349)
0928 13:02:04.307793 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:04.307844 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260928 13:02:04.787061 28258 raft_consensus.cc:482] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:04.787185 28258 raft_consensus.cc:504] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39664385a0f1451a90a11ab2076ef5ef" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 39765 } } peers { permanent_uuid: "69ee8725ef1a4a5898165a56837c7b18" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 42743 } } peers { permanent_uuid: "fdcfbe72b22a49d4b6346f8f0828653c" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38537 } }
I20260928 13:02:04.787443 28258 leader_election.cc:288] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765), 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743)
W20260928 13:02:04.787827 27844 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ee8725ef1a4a5898165a56837c7b18 (127.11.11.67:42743): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
W20260928 13:02:04.787896 27842 leader_election.cc:341] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 39664385a0f1451a90a11ab2076ef5ef (127.11.11.66:39765): Not found: Tablet not found: 2b07e2449836466faea7416421ead4cf
I20260928 13:02:04.787950 27842 leader_election.cc:302] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdcfbe72b22a49d4b6346f8f0828653c; no voters: 39664385a0f1451a90a11ab2076ef5ef, 69ee8725ef1a4a5898165a56837c7b18
I20260928 13:02:04.788116 28258 raft_consensus.cc:2726] T 2b07e2449836466faea7416421ead4cf P fdcfbe72b22a49d4b6346f8f0828653c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:02:05.101550 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:05.101268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:54867)
0928 13:02:05.101442 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:05.101496 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260928 13:02:05.175648 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:05.175360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:43259)
0928 13:02:05.175503 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:05.175600 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 13:02:05.308723 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:05.308442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:44015)
0928 13:02:05.308638 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:05.308677 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":113,"threads_started":1}
I20260928 13:02:06.102388 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:06.102137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:35583)
0928 13:02:06.102297 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:06.102339 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260928 13:02:06.176446 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:06.176212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:55915)
0928 13:02:06.176357 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:06.176396 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260928 13:02:06.309525 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:06.309285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:36209)
0928 13:02:06.309426 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:06.309475 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260928 13:02:07.103211 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:07.102973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:52811)
0928 13:02:07.103115 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:07.103161 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260928 13:02:07.177237 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:07.176936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:44163)
0928 13:02:07.177139 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:07.177188 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260928 13:02:07.310266 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:07.310039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:34749)
0928 13:02:07.310180 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:07.310219 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 13:02:08.104135 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:08.103831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:59679)
0928 13:02:08.103972 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:08.104013 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 13:02:08.177973 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:08.177739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:34435)
0928 13:02:08.177885 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:08.177924 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260928 13:02:08.311023 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:08.310786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:45341)
0928 13:02:08.310935 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:08.310976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260928 13:02:09.104923 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:09.104654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:37595)
0928 13:02:09.104829 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:09.104875 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 13:02:09.178745 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:09.178516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:46033)
0928 13:02:09.178654 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:09.178695 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260928 13:02:09.311872 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:09.311632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:48515)
0928 13:02:09.311783 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:09.311823 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260928 13:02:10.105741 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:10.105518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.65:51871)
0928 13:02:10.105660 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:10.105692 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260928 13:02:10.179695 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:10.179362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.66:53379)
0928 13:02:10.179513 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:10.179645 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 13:02:10.306862 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27791
I20260928 13:02:10.311717 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 27959
I20260928 13:02:10.313279 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:10.312981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46085 (local address 127.11.11.67:44839)
0928 13:02:10.313177 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:10.313220 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.126:46085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":55,"threads_started":1}
I20260928 13:02:10.316359 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28092
2026-09-28T13:02:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58309 ms)
[----------] 30 tests from SecurityITest (239970 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:10 dist-test-slave-8sjz krb5kdc[28279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:10 dist-test-slave-8sjz krb5kdc[28279](info): set up 2 sockets
Sep 28 13:02:10 dist-test-slave-8sjz krb5kdc[28279](info): commencing operation
krb5kdc: starting...
W20260928 13:02:11.384917 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:11.512255 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35879
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:11.595969 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:11.596238 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:11.596263 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:11.597589 28292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:11.597642 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:11.597657 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:11.597671 28292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:11.597708 28292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:11.599294 28292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:35879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:11.599567 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:11.599826 28292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:11.605222 28300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:11.605250 28298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:11.605222 28297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:11.605242 28292 server_base.cc:1034] running on GCE node
I20260928 13:02:11.605795 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:11.606114 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:11.607292 28292 hybrid_clock.cc:648] HybridClock initialized: now 1790600531607262 us; error 40 us; skew 500 ppm
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:11.610359 28292 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:11.610762 28292 webserver.cc:460] Webserver started at http://127.11.11.126:39043/ using document root <none> and password file <none>
I20260928 13:02:11.610968 28292 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:11.611011 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:11.611102 28292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:11.612234 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "522de3717304436dbeafe72fa4ac3f72"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "0cd5449671e2cf74623e9b594e00227c"
server_key_iv: "4d996bf6067a07df3e9b48a7f66fc2e3"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.612550 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "522de3717304436dbeafe72fa4ac3f72"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "0cd5449671e2cf74623e9b594e00227c"
server_key_iv: "4d996bf6067a07df3e9b48a7f66fc2e3"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.613865 28292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:11.614629 28307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.614888 28292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:02:11.614966 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "522de3717304436dbeafe72fa4ac3f72"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "0cd5449671e2cf74623e9b594e00227c"
server_key_iv: "4d996bf6067a07df3e9b48a7f66fc2e3"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.615058 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:11.631486 28292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:11.634078 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:11.634259 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:11.638387 28292 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:35879
I20260928 13:02:11.638423 28359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:35879 every 8 connection(s)
I20260928 13:02:11.638804 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:11.639719 28360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:11.642132 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: Bootstrap starting.
I20260928 13:02:11.642746 28360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:11.643051 28360 log.cc:824] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:11.643847 28360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: No bootstrap required, opened a new log
I20260928 13:02:11.645629 28360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } }
I20260928 13:02:11.645778 28360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:11.645802 28360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522de3717304436dbeafe72fa4ac3f72, State: Initialized, Role: FOLLOWER
I20260928 13:02:11.645957 28360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } }
I20260928 13:02:11.646039 28360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:11.646061 28360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:11.646126 28360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:11.646719 28360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } }
I20260928 13:02:11.646833 28360 leader_election.cc:302] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 522de3717304436dbeafe72fa4ac3f72; no voters:
I20260928 13:02:11.647047 28360 leader_election.cc:288] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:11.647154 28363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:11.647336 28360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:11.647630 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28292
I20260928 13:02:11.647727 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:11.647940 11309 external_mini_cluster.cc:1428] Setting key 26ff6ebc5bc8e55e4814b173642a0856
I20260928 13:02:11.647339 28363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [term 1 LEADER]: Becoming Leader. State: Replica: 522de3717304436dbeafe72fa4ac3f72, State: Running, Role: LEADER
I20260928 13:02:11.648263 28363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } }
I20260928 13:02:11.649201 28364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "522de3717304436dbeafe72fa4ac3f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } } }
I20260928 13:02:11.649303 28364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:11.649627 28365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 522de3717304436dbeafe72fa4ac3f72. Latest consensus state: current_term: 1 leader_uuid: "522de3717304436dbeafe72fa4ac3f72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "522de3717304436dbeafe72fa4ac3f72" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 35879 } } }
I20260928 13:02:11.649775 28365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:11.649823 28375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:11.650516 28375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:11.652684 28375 catalog_manager.cc:1347] Generated new cluster ID: ccd468a724ff4b65b3db061c86c2e974
I20260928 13:02:11.652742 28375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:11.658979 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:11.648437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.11.11.126:35879)
0928 13:02:11.648604 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:11.648616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:11.648674 (+ 58us) server_negotiation.cc:407] Connection header received
0928 13:02:11.648820 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:11.648829 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:11.648900 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:11.649931 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:11.650059 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.651447 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.652310 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.652924 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.653051 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:11.655493 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:11.655498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:11.655498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:11.655517 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:11.657325 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:11.657736 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:11.657739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:11.657739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:11.657853 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:11.658070 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:11.658071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:11.658072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:11.658349 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:11.658550 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:11.658633 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:02:11.658794 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260928 13:02:11.662542 28375 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:11.663280 28375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:11.668248 28375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: Generated new TSK 0
I20260928 13:02:11.668453 28375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:02:11.703594 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:11.787807 28387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:11.788094 28387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:11.788141 28387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:11.789389 28387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:11.789433 28387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:11.789461 28387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:11.790906 28387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:11.791114 28387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:11.791301 28387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:11.796861 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:11.796810 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:11.797015 28387 server_base.cc:1034] running on GCE node
W20260928 13:02:11.796808 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:11.797250 28387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:11.797497 28387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:11.798656 28387 hybrid_clock.cc:648] HybridClock initialized: now 1790600531798639 us; error 40 us; skew 500 ppm
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:11.801235 28387 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:02:11.801662 28387 webserver.cc:460] Webserver started at http://127.11.11.65:34089/ using document root <none> and password file <none>
I20260928 13:02:11.801898 28387 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:11.801965 28387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:11.802112 28387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:11.803161 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "423ee5a99f704237a5beea980acee8f5"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "9c7c89c81dab3916fe59603ae3274f8c"
server_key_iv: "f82bf11ff043226a216cd4ae99732c2d"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.803527 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "423ee5a99f704237a5beea980acee8f5"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "9c7c89c81dab3916fe59603ae3274f8c"
server_key_iv: "f82bf11ff043226a216cd4ae99732c2d"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.804961 28387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:11.805720 28402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.805891 28387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.806003 28387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "423ee5a99f704237a5beea980acee8f5"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "9c7c89c81dab3916fe59603ae3274f8c"
server_key_iv: "f82bf11ff043226a216cd4ae99732c2d"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.806077 28387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:11.817637 28387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:11.819916 28387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:11.820145 28387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:11.820398 28387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:11.820713 28387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:11.820768 28387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.820808 28387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:11.820834 28387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.826731 28387 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:37223
I20260928 13:02:11.826782 28515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:37223 every 8 connection(s)
I20260928 13:02:11.827144 28387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:02:11.828861 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28387
I20260928 13:02:11.828979 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:11.829213 11309 external_mini_cluster.cc:1428] Setting key b656a3e23781133cd4734a10c90d65a6
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 13:02:11.837718 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:11.827906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:33265 (local address 127.11.11.126:35879)
0928 13:02:11.827951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:11.827954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:11.828210 (+ 256us) server_negotiation.cc:407] Connection header received
0928 13:02:11.828831 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:11.828832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:11.828848 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:11.828901 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:11.830247 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.830805 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.831984 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.832275 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.832365 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:11.834603 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:11.834604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:11.834604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:11.834612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:11.836221 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:11.836658 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:11.836659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:11.836660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:11.836723 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:11.837020 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:11.837021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:11.837021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:11.837149 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:11.837227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:11.837522 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 13:02:11.837614 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:11.837911 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:11.827870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35879 (local address 127.11.11.65:33265)
0928 13:02:11.828013 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:11.828110 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:11.828701 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:11.828987 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:11.828993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:11.829813 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:11.830123 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:11.830130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.830910 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.830911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:11.831885 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:11.831887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:11.832377 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:11.832378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:11.832525 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:11.833107 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:11.833116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:11.834493 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:11.836320 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:11.836328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:11.836330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:11.836572 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:11.836794 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:11.836794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:11.836795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:11.836920 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:11.837222 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:11.837234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:11.837387 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 13:02:11.837500 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:02:11.837667 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260928 13:02:11.838511 28516 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35879
I20260928 13:02:11.838621 28516 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:11.838800 28516 heartbeater.cc:511] Master 127.11.11.126:35879 requested a full tablet report, sending...
I20260928 13:02:11.839389 28324 ts_manager.cc:194] Registered new tserver with Master: 423ee5a99f704237a5beea980acee8f5 (127.11.11.65:37223)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
I20260928 13:02:11.840191 28324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:33265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:02:11.855543 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:11.945990 28521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:11.946214 28521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:11.946239 28521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:11.947379 28521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:11.947417 28521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:11.947443 28521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:11.949098 28521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:11.949301 28521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:11.949537 28521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:11.955191 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:11.955140 28526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:11.955156 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:11.955256 28521 server_base.cc:1034] running on GCE node
I20260928 13:02:11.955682 28521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:11.955962 28521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:11.957142 28521 hybrid_clock.cc:648] HybridClock initialized: now 1790600531957150 us; error 55 us; skew 500 ppm
Sep 28 13:02:11 dist-test-slave-8sjz krb5kdc[28279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:11.959919 28521 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:02:11.960417 28521 webserver.cc:460] Webserver started at http://127.11.11.66:46321/ using document root <none> and password file <none>
I20260928 13:02:11.960721 28521 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:11.960793 28521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:11.960933 28521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:11.962080 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "2a2d4a075f2042999c486f859ad52d0b"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "c707cb6143a2394462dc9cc6d0c4a75f"
server_key_iv: "84c5f89720ee59b336632eaa84b4c825"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.962460 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "2a2d4a075f2042999c486f859ad52d0b"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "c707cb6143a2394462dc9cc6d0c4a75f"
server_key_iv: "84c5f89720ee59b336632eaa84b4c825"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.963960 28521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:11.964890 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.965118 28521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:11.965230 28521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "2a2d4a075f2042999c486f859ad52d0b"
format_stamp: "Formatted at 2026-09-28 13:02:11 on dist-test-slave-8sjz"
server_key: "c707cb6143a2394462dc9cc6d0c4a75f"
server_key_iv: "84c5f89720ee59b336632eaa84b4c825"
server_key_version: "encryptionkey@0"
I20260928 13:02:11.965324 28521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:11.989586 28521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:11.991822 28521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:11.991997 28521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:11.992288 28521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:11.992609 28521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:11.992664 28521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.992765 28521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:11.992796 28521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:11.999137 28521 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:40881
I20260928 13:02:11.999194 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:40881 every 8 connection(s)
I20260928 13:02:11.999513 28521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:12.001334 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28521
I20260928 13:02:12.001443 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:12.001641 11309 external_mini_cluster.cc:1428] Setting key ed2de14b6988136e48f6b6ecfaee8d75
Sep 28 13:02:12 dist-test-slave-8sjz krb5kdc[28279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600531, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 13:02:12.010432 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.000278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:41059 (local address 127.11.11.126:35879)
0928 13:02:12.000325 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.000328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.000551 (+ 223us) server_negotiation.cc:407] Connection header received
0928 13:02:12.001108 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.001110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.001124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.001168 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.002432 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.002935 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.004335 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.004529 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.004590 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.007271 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.007272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.007272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:12.007280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.008771 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.009334 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.009335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.009335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.009388 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.009707 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.009707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.009708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.009809 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.009861 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.010277 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.010363 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:12.010656 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.000258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35879 (local address 127.11.11.66:41059)
0928 13:02:12.000416 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.000462 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.000998 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.001263 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.001275 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.002043 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:12.002327 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.002334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.003032 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.003033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.004248 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.004251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.004619 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.004620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.004773 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.005349 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:12.005357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:12.007170 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:12.008910 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:12.008917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:12.008919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:12.009256 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:12.009461 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:12.009462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:12.009462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:12.009613 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:12.009881 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:12.009890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:12.010081 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.010220 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.010410 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260928 13:02:12.011197 28650 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35879
I20260928 13:02:12.011314 28650 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:12.011521 28650 heartbeater.cc:511] Master 127.11.11.126:35879 requested a full tablet report, sending...
I20260928 13:02:12.011876 28324 ts_manager.cc:194] Registered new tserver with Master: 2a2d4a075f2042999c486f859ad52d0b (127.11.11.66:40881)
I20260928 13:02:12.012327 28324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:41059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:02:12.030712 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:12.113296 28655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:12.113529 28655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:12.113555 28655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:12.114883 28655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:12.114928 28655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:12.114956 28655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:12.116575 28655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:35879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:12.116832 28655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:12.117067 28655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:02:12.122346 28655 server_base.cc:1034] running on GCE node
W20260928 13:02:12.122318 28660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.122325 28663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.122357 28661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.122766 28655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:12.123027 28655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:12.124214 28655 hybrid_clock.cc:648] HybridClock initialized: now 1790600532124193 us; error 32 us; skew 500 ppm
Sep 28 13:02:12 dist-test-slave-8sjz krb5kdc[28279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600532, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:12.126827 28655 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:02:12.127290 28655 webserver.cc:460] Webserver started at http://127.11.11.67:41539/ using document root <none> and password file <none>
I20260928 13:02:12.127492 28655 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:12.127533 28655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:12.127629 28655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:12.128700 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "6a502198c5c74baca8495a0186e4125c"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "6b10f0ad483e4d4c1b3f57fa07ae7cf2"
server_key_iv: "3cc17a7fd78a0affebb16987f10a4fe0"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.129004 28655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "6a502198c5c74baca8495a0186e4125c"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "6b10f0ad483e4d4c1b3f57fa07ae7cf2"
server_key_iv: "3cc17a7fd78a0affebb16987f10a4fe0"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.130275 28655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:12.131090 28670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.131301 28655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:12.131398 28655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "6a502198c5c74baca8495a0186e4125c"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "6b10f0ad483e4d4c1b3f57fa07ae7cf2"
server_key_iv: "3cc17a7fd78a0affebb16987f10a4fe0"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.131456 28655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:12.156539 28655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:12.158564 28655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:12.158692 28655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:12.158926 28655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:12.159317 28655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:12.159355 28655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.159379 28655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:12.159391 28655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.165364 28655 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40059
I20260928 13:02:12.165506 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40059 every 8 connection(s)
I20260928 13:02:12.165771 28655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:02:12.166128 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28655
I20260928 13:02:12.166217 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:12.166397 11309 external_mini_cluster.cc:1428] Setting key 413ada876214676631157dd02d8456d8
Sep 28 13:02:12 dist-test-slave-8sjz krb5kdc[28279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600532, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:12.175613 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.166526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:60845 (local address 127.11.11.126:35879)
0928 13:02:12.166574 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.166576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.166907 (+ 331us) server_negotiation.cc:407] Connection header received
0928 13:02:12.167582 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.167583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.167604 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.167661 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.168879 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.169488 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.170524 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.170760 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.170835 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.173074 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.173075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.173075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:12.173082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.174416 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.174834 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.174835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.174835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.174885 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.175095 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.175095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.175096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.175200 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.175251 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.175484 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.175547 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:12.175908 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.166527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:35879 (local address 127.11.11.67:60845)
0928 13:02:12.166744 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.166797 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.167477 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.167736 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.167740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.168553 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:12.168777 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.168782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.169582 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.169584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.170424 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.170426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.170854 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.170855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.170995 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.171556 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:12.171565 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:12.172988 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:12.174491 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:12.174497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:12.174498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:12.174767 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:12.174944 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:12.174944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:12.174944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:12.175040 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:12.175255 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:12.175259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:12.175373 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.175469 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.175592 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 13:02:12.176477 28784 heartbeater.cc:347] Connected to a master server at 127.11.11.126:35879
I20260928 13:02:12.176591 28784 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:12.176848 28784 heartbeater.cc:511] Master 127.11.11.126:35879 requested a full tablet report, sending...
I20260928 13:02:12.177233 28324 ts_manager.cc:194] Registered new tserver with Master: 6a502198c5c74baca8495a0186e4125c (127.11.11.67:40059)
I20260928 13:02:12.177706 28324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:60845
I20260928 13:02:12.178203 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:12.185382 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.179524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.11.11.126:35879)
0928 13:02:12.179596 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.179598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.179695 (+ 97us) server_negotiation.cc:407] Connection header received
0928 13:02:12.179730 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.179731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.179739 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.179784 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.180355 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.180827 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.181488 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.181659 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.181723 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.182943 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.182944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.182944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:12.182950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.184280 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.184609 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.184610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.184610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.184659 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:12.184893 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:12.184894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:12.184894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:12.185030 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.185104 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.185227 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.185287 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:02:12.187075 28324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:12.188782 28324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:12.195606 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.11.11.66:40881)
0928 13:02:12.191857 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.191862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.191874 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:02:12.191916 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.191918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.191935 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.192217 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.192297 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.193528 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.194637 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195127 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195233 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.195467 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.195513 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 13:02:12.195616 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:40881 (local address 127.0.0.1:46012)
0928 13:02:12.191472 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.191483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.191537 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.192070 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.192071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.192073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.192124 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.192126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.193633 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.193634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.194535 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.194537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195238 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.195340 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195353 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.195444 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.195513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":24,"thread_start_us":100,"threads_started":1}
I20260928 13:02:12.195619 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:37223 (local address 127.0.0.1:51270)
0928 13:02:12.191625 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.191635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.191675 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.191866 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.191870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.191872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.192022 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.192024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.193059 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.193068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.194266 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.194269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195300 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.195365 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195369 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.195423 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.195509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":86,"threads_started":1}
I20260928 13:02:12.195947 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58320 (local address 127.11.11.67:40059)
0928 13:02:12.191926 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.191930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.191940 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:12.191976 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.191978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.191993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.192091 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.192298 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.193376 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.194671 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195419 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195530 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.195821 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.195862 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":86,"threads_started":1}
I20260928 13:02:12.195969 28795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40059 (local address 127.0.0.1:58320)
0928 13:02:12.191376 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.191417 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.191481 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.192176 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.192177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.192179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.192219 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.192220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.193487 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.193488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.194581 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.194584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195579 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.195643 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195646 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.195697 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.195781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":22,"thread_start_us":55,"threads_started":1}
I20260928 13:02:12.196507 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.191474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.11.11.65:37223)
0928 13:02:12.191684 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.191690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.191701 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:12.191743 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.191747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.191767 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.191859 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.192129 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.192948 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.194366 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.195128 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.195199 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.195261 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.196377 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.196433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260928 13:02:12.196574 28584 tablet_service.cc:1467] Processing CreateTablet for tablet 281021a6291e4d07a73708143b3fb1ec (DEFAULT_TABLE table=test-table [id=88b5e019023348d0a56e6135bca162fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:12.196738 28718 tablet_service.cc:1467] Processing CreateTablet for tablet 281021a6291e4d07a73708143b3fb1ec (DEFAULT_TABLE table=test-table [id=88b5e019023348d0a56e6135bca162fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:12.196879 28584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281021a6291e4d07a73708143b3fb1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.197062 28718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281021a6291e4d07a73708143b3fb1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.197206 28450 tablet_service.cc:1467] Processing CreateTablet for tablet 281021a6291e4d07a73708143b3fb1ec (DEFAULT_TABLE table=test-table [id=88b5e019023348d0a56e6135bca162fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:12.197479 28450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281021a6291e4d07a73708143b3fb1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.199751 28801 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Bootstrap starting.
I20260928 13:02:12.200413 28802 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Bootstrap starting.
I20260928 13:02:12.200485 28801 tablet_bootstrap.cc:654] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:12.200757 28803 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Bootstrap starting.
I20260928 13:02:12.201071 28801 log.cc:824] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:12.201203 28802 tablet_bootstrap.cc:654] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:12.201390 28803 tablet_bootstrap.cc:654] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:12.201761 28802 log.cc:824] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:12.201942 28801 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: No bootstrap required, opened a new log
I20260928 13:02:12.202045 28801 ts_tablet_manager.cc:1397] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:02:12.202102 28803 log.cc:824] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:12.202554 28802 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: No bootstrap required, opened a new log
I20260928 13:02:12.202690 28802 ts_tablet_manager.cc:1397] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:02:12.202875 28803 tablet_bootstrap.cc:492] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: No bootstrap required, opened a new log
I20260928 13:02:12.202958 28803 ts_tablet_manager.cc:1397] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:02:12.204255 28801 raft_consensus.cc:348] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.204449 28801 raft_consensus.cc:374] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:12.204485 28801 raft_consensus.cc:729] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2d4a075f2042999c486f859ad52d0b, State: Initialized, Role: FOLLOWER
I20260928 13:02:12.204496 28803 raft_consensus.cc:348] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.204615 28801 consensus_queue.cc:260] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.204703 28803 raft_consensus.cc:374] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:12.204751 28803 raft_consensus.cc:729] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423ee5a99f704237a5beea980acee8f5, State: Initialized, Role: FOLLOWER
I20260928 13:02:12.204721 28802 raft_consensus.cc:348] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.204890 28802 raft_consensus.cc:374] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:12.204855 28803 consensus_queue.cc:260] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.204926 28801 ts_tablet_manager.cc:1428] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:02:12.204934 28802 raft_consensus.cc:729] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a502198c5c74baca8495a0186e4125c, State: Initialized, Role: FOLLOWER
I20260928 13:02:12.205058 28650 heartbeater.cc:503] Master 127.11.11.126:35879 was elected leader, sending a full tablet report...
I20260928 13:02:12.205040 28802 consensus_queue.cc:260] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.205143 28803 ts_tablet_manager.cc:1428] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:02:12.205258 28516 heartbeater.cc:503] Master 127.11.11.126:35879 was elected leader, sending a full tablet report...
I20260928 13:02:12.205322 28802 ts_tablet_manager.cc:1428] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:12.205432 28784 heartbeater.cc:503] Master 127.11.11.126:35879 was elected leader, sending a full tablet report...
W20260928 13:02:12.250172 28651 tablet.cc:2369] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:12.303895 28807 raft_consensus.cc:482] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:12.304091 28807 raft_consensus.cc:504] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.304540 28807 leader_election.cc:288] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a502198c5c74baca8495a0186e4125c (127.11.11.67:40059), 423ee5a99f704237a5beea980acee8f5 (127.11.11.65:37223)
I20260928 13:02:12.308560 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.304803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:43791 (local address 127.11.11.65:37223)
0928 13:02:12.304848 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.304850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.304863 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:02:12.304934 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.304935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.304946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.305020 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.305200 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.306014 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.307581 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.308183 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.308264 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.308272 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.308446 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.308502 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:02:12.308559 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.304894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:40785 (local address 127.11.11.67:40059)
0928 13:02:12.304953 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.304955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.305063 (+ 108us) server_negotiation.cc:407] Connection header received
0928 13:02:12.305094 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.305095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.305108 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.305151 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.305304 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.306012 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.307583 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.308190 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.308264 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.308272 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.308446 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.308485 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:02:12.308656 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.304738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:37223 (local address 127.11.11.66:43791)
0928 13:02:12.304797 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.304811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.304864 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.305042 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.305043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.305046 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.305122 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.305124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.306105 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.306106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.307483 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.307487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.308270 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.308271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.308358 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.308359 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.308424 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.308468 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":70}
I20260928 13:02:12.309023 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.304781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40059 (local address 127.11.11.66:40785)
0928 13:02:12.304986 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.304996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.305048 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.305171 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.305172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.305173 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.305240 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.305241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.306113 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.306114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.307483 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.307487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.308270 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.308271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.308353 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.308358 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.308422 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.308468 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":63,"thread_start_us":146,"threads_started":1}
I20260928 13:02:12.309038 28470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281021a6291e4d07a73708143b3fb1ec" candidate_uuid: "2a2d4a075f2042999c486f859ad52d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "423ee5a99f704237a5beea980acee8f5" is_pre_election: true
I20260928 13:02:12.309221 28470 raft_consensus.cc:2454] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a2d4a075f2042999c486f859ad52d0b in term 0.
I20260928 13:02:12.309427 28738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281021a6291e4d07a73708143b3fb1ec" candidate_uuid: "2a2d4a075f2042999c486f859ad52d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a502198c5c74baca8495a0186e4125c" is_pre_election: true
I20260928 13:02:12.309450 28538 leader_election.cc:302] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a2d4a075f2042999c486f859ad52d0b, 423ee5a99f704237a5beea980acee8f5; no voters:
I20260928 13:02:12.309542 28738 raft_consensus.cc:2454] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a2d4a075f2042999c486f859ad52d0b in term 0.
I20260928 13:02:12.309602 28807 raft_consensus.cc:2781] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:12.309661 28807 raft_consensus.cc:482] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:12.309829 28807 raft_consensus.cc:3037] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:12.310567 28807 raft_consensus.cc:504] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.310730 28807 leader_election.cc:288] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [CANDIDATE]: Term 1 election: Requested vote from peers 6a502198c5c74baca8495a0186e4125c (127.11.11.67:40059), 423ee5a99f704237a5beea980acee8f5 (127.11.11.65:37223)
I20260928 13:02:12.310905 28738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281021a6291e4d07a73708143b3fb1ec" candidate_uuid: "2a2d4a075f2042999c486f859ad52d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a502198c5c74baca8495a0186e4125c"
I20260928 13:02:12.310905 28470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281021a6291e4d07a73708143b3fb1ec" candidate_uuid: "2a2d4a075f2042999c486f859ad52d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "423ee5a99f704237a5beea980acee8f5"
I20260928 13:02:12.310990 28470 raft_consensus.cc:3037] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:12.310990 28738 raft_consensus.cc:3037] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:12.311909 28470 raft_consensus.cc:2454] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a2d4a075f2042999c486f859ad52d0b in term 1.
I20260928 13:02:12.311909 28738 raft_consensus.cc:2454] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a2d4a075f2042999c486f859ad52d0b in term 1.
I20260928 13:02:12.312115 28537 leader_election.cc:302] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a2d4a075f2042999c486f859ad52d0b, 6a502198c5c74baca8495a0186e4125c; no voters:
I20260928 13:02:12.312206 28807 raft_consensus.cc:2781] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:12.312353 28807 raft_consensus.cc:686] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 1 LEADER]: Becoming Leader. State: Replica: 2a2d4a075f2042999c486f859ad52d0b, State: Running, Role: LEADER
I20260928 13:02:12.312462 28807 consensus_queue.cc:237] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } }
I20260928 13:02:12.313437 28322 catalog_manager.cc:5495] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a2d4a075f2042999c486f859ad52d0b (127.11.11.66). New cstate: current_term: 1 leader_uuid: "2a2d4a075f2042999c486f859ad52d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a2d4a075f2042999c486f859ad52d0b" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 40881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 } health_report { overall_health: UNKNOWN } } }
W20260928 13:02:12.327855 28517 tablet.cc:2369] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:12.340899 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.338117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.11.11.66:40881)
0928 13:02:12.338187 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.338189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.338200 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:12.338252 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.338253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.338271 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.338356 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:12.338538 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.339034 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.339998 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.340237 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.340317 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.340513 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:12.340629 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:12.340728 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.340747 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.340778 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:02:12.342928 28470 raft_consensus.cc:1261] T 281021a6291e4d07a73708143b3fb1ec P 423ee5a99f704237a5beea980acee8f5 [term 1 FOLLOWER]: Refusing update from remote peer 2a2d4a075f2042999c486f859ad52d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:12.342972 28738 raft_consensus.cc:1261] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 1 FOLLOWER]: Refusing update from remote peer 2a2d4a075f2042999c486f859ad52d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:12.343376 28811 consensus_queue.cc:1035] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "423ee5a99f704237a5beea980acee8f5" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 37223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:12.343573 28807 consensus_queue.cc:1035] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a502198c5c74baca8495a0186e4125c" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:12.346846 28815 mvcc.cc:205] Tried to move back new op lower bound from 7334299780473597952 to 7334299780352876544. Current Snapshot: MvccSnapshot[applied={T|T < 7334299780473597952 or (T in {7334299780473597952})}]
I20260928 13:02:12.346900 28816 mvcc.cc:205] Tried to move back new op lower bound from 7334299780473597952 to 7334299780352876544. Current Snapshot: MvccSnapshot[applied={T|T < 7334299780473597952}]
I20260928 13:02:12.349614 28322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:12.349751 28322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:12.350613 28322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 522de3717304436dbeafe72fa4ac3f72: Sending DeleteTablet for 3 replicas of tablet 281021a6291e4d07a73708143b3fb1ec
I20260928 13:02:12.351047 28584 tablet_service.cc:1514] Processing DeleteTablet for tablet 281021a6291e4d07a73708143b3fb1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:12 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:46012
I20260928 13:02:12.351089 28450 tablet_service.cc:1514] Processing DeleteTablet for tablet 281021a6291e4d07a73708143b3fb1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:12 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:51270
I20260928 13:02:12.351282 28824 tablet_replica.cc:330] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: stopping tablet replica
I20260928 13:02:12.351377 28824 raft_consensus.cc:2229] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:02:12.351450 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28387
I20260928 13:02:12.351469 28718 tablet_service.cc:1514] Processing DeleteTablet for tablet 281021a6291e4d07a73708143b3fb1ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:12 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:58320
I20260928 13:02:12.351507 28824 raft_consensus.cc:2258] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:12.352901 28824 ts_tablet_manager.cc:1905] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:12.355002 28824 ts_tablet_manager.cc:1918] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:12.355180 28824 log.cc:1196] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/281021a6291e4d07a73708143b3fb1ec
I20260928 13:02:12.355641 28824 ts_tablet_manager.cc:1939] T 281021a6291e4d07a73708143b3fb1ec P 2a2d4a075f2042999c486f859ad52d0b: Deleting consensus metadata
I20260928 13:02:12.356220 28826 tablet_replica.cc:330] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: stopping tablet replica
I20260928 13:02:12.356383 28826 raft_consensus.cc:2229] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:12.356535 28826 raft_consensus.cc:2258] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:12.356994 28826 ts_tablet_manager.cc:1905] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 13:02:12.357607 28309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:37223 (error 108)
I20260928 13:02:12.358003 28311 catalog_manager.cc:4841] TS 2a2d4a075f2042999c486f859ad52d0b (127.11.11.66:40881): tablet 281021a6291e4d07a73708143b3fb1ec (table test-table [id=88b5e019023348d0a56e6135bca162fe]) successfully deleted
I20260928 13:02:12.358390 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28521
I20260928 13:02:12.359246 28826 ts_tablet_manager.cc:1918] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:12.359321 28826 log.cc:1196] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/281021a6291e4d07a73708143b3fb1ec
I20260928 13:02:12.359453 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.359297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:37223 (local address 127.0.0.1:51284)
0928 13:02:12.359368 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.359412 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:37223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 13:02:12.359591 28309 catalog_manager.cc:4584] TS 423ee5a99f704237a5beea980acee8f5 (127.11.11.65:37223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 281021a6291e4d07a73708143b3fb1ec: Network error: Client connection negotiation failed: client connection to 127.11.11.65:37223: connect: Connection refused (error 111)
I20260928 13:02:12.359625 28826 ts_tablet_manager.cc:1939] T 281021a6291e4d07a73708143b3fb1ec P 6a502198c5c74baca8495a0186e4125c: Deleting consensus metadata
I20260928 13:02:12.360087 28308 catalog_manager.cc:4841] TS 6a502198c5c74baca8495a0186e4125c (127.11.11.67:40059): tablet 281021a6291e4d07a73708143b3fb1ec (table test-table [id=88b5e019023348d0a56e6135bca162fe]) successfully deleted
I20260928 13:02:12.364403 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28655
I20260928 13:02:12.369549 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28292
2026-09-28T13:02:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T13:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:12Z Disabled control of system clock
I20260928 13:02:12.398501 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38221
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:38221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:12.479913 28831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:12.480250 28831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:12.480291 28831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:12.481995 28831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:12.482122 28831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:12.482153 28831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:12.482178 28831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:12.482213 28831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:12.483942 28831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:38221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:12.484329 28831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:12.484735 28831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:12.490329 28836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.490274 28837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.490382 28839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.490437 28831 server_base.cc:1034] running on GCE node
I20260928 13:02:12.490825 28831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:12.491202 28831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:12.492359 28831 hybrid_clock.cc:648] HybridClock initialized: now 1790600532492348 us; error 30 us; skew 500 ppm
I20260928 13:02:12.493835 28831 webserver.cc:460] Webserver started at http://127.11.11.126:38479/ using document root <none> and password file <none>
I20260928 13:02:12.494114 28831 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:12.494166 28831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:12.494354 28831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:12.495479 28831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "28d14167e8a7485bbe42e5bb4b34da12"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "14f19caf5efe84e2b6a8f8801a04fd7d"
server_key_iv: "54c347faac345fe1980d42183043b7d9"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.495867 28831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "28d14167e8a7485bbe42e5bb4b34da12"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "14f19caf5efe84e2b6a8f8801a04fd7d"
server_key_iv: "54c347faac345fe1980d42183043b7d9"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.497252 28831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:12.498072 28845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.498255 28831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:12.498363 28831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "28d14167e8a7485bbe42e5bb4b34da12"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "14f19caf5efe84e2b6a8f8801a04fd7d"
server_key_iv: "54c347faac345fe1980d42183043b7d9"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.498463 28831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:12.532636 28831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:12.533015 28831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:12.533190 28831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:12.537401 28831 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:38221
I20260928 13:02:12.537443 28897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:38221 every 8 connection(s)
I20260928 13:02:12.537839 28831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:12.538859 28898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.541245 28898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: Bootstrap starting.
I20260928 13:02:12.541853 28898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:12.542179 28898 log.cc:824] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:12.542876 28898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: No bootstrap required, opened a new log
I20260928 13:02:12.543460 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28831
I20260928 13:02:12.543578 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:12.543779 11309 external_mini_cluster.cc:1428] Setting key 3edbb68574d4aec89c82d2aa302ed757
I20260928 13:02:12.545234 28898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } }
I20260928 13:02:12.545413 28898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:12.545452 28898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d14167e8a7485bbe42e5bb4b34da12, State: Initialized, Role: FOLLOWER
I20260928 13:02:12.545632 28898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } }
I20260928 13:02:12.545733 28898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:12.545759 28898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:12.545830 28898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:12.546391 28898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } }
I20260928 13:02:12.546476 28898 leader_election.cc:302] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d14167e8a7485bbe42e5bb4b34da12; no voters:
I20260928 13:02:12.546689 28898 leader_election.cc:288] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:12.546720 28903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:12.546829 28903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [term 1 LEADER]: Becoming Leader. State: Replica: 28d14167e8a7485bbe42e5bb4b34da12, State: Running, Role: LEADER
I20260928 13:02:12.546977 28898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:12.546972 28903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } }
I20260928 13:02:12.547508 28904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28d14167e8a7485bbe42e5bb4b34da12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } } }
I20260928 13:02:12.547797 28904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:12.547722 28905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d14167e8a7485bbe42e5bb4b34da12. Latest consensus state: current_term: 1 leader_uuid: "28d14167e8a7485bbe42e5bb4b34da12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d14167e8a7485bbe42e5bb4b34da12" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 38221 } } }
I20260928 13:02:12.548156 28905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:12.548460 28912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:12.548944 28912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:12.550938 28912 catalog_manager.cc:1347] Generated new cluster ID: e142149f6e1e42129a8695bc898314b9
I20260928 13:02:12.551003 28912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:12.551056 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.544390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.11.11.126:38221)
0928 13:02:12.544640 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.544648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.544678 (+ 30us) server_negotiation.cc:407] Connection header received
0928 13:02:12.545217 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.545224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.545286 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.545427 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.545687 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.547222 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.548385 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.548992 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.549114 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.550113 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.550117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.550118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:12.550364 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.550416 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:12.550434 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.550506 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.550728 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.550811 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260928 13:02:12.585611 28912 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:12.586383 28912 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:12.597051 28912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: Generated new TSK 0
I20260928 13:02:12.597309 28912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:12.600200 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--builtin_ntp_servers=127.11.11.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:12.683538 28922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:12.683786 28922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:12.683810 28922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:12.685050 28922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:12.685096 28922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:12.685122 28922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:12.686507 28922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:12.686790 28922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:12.687108 28922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:12.692399 28927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.692402 28930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.692399 28928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.692687 28922 server_base.cc:1034] running on GCE node
I20260928 13:02:12.692936 28922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:12.693189 28922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:12.694351 28922 hybrid_clock.cc:648] HybridClock initialized: now 1790600532694340 us; error 31 us; skew 500 ppm
I20260928 13:02:12.695644 28922 webserver.cc:460] Webserver started at http://127.11.11.65:43829/ using document root <none> and password file <none>
I20260928 13:02:12.695909 28922 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:12.695981 28922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:12.696162 28922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:12.697268 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "9bf78fcd5afc4cb2b116a489a54c16d9"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "048f4648b14dcac633c91b9ff6da3031"
server_key_iv: "34b93e0048ef0ebf03ec06782b2825ec"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.697643 28922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "9bf78fcd5afc4cb2b116a489a54c16d9"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "048f4648b14dcac633c91b9ff6da3031"
server_key_iv: "34b93e0048ef0ebf03ec06782b2825ec"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.698987 28922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:12.699715 28936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.699894 28922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:02:12.699994 28922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "9bf78fcd5afc4cb2b116a489a54c16d9"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "048f4648b14dcac633c91b9ff6da3031"
server_key_iv: "34b93e0048ef0ebf03ec06782b2825ec"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.700107 28922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:12.712474 28922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:12.712822 28922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:12.712982 28922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:12.713229 28922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:12.713580 28922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:12.713637 28922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.713681 28922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:12.713711 28922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.719575 28922 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:41751
I20260928 13:02:12.719638 29049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:41751 every 8 connection(s)
I20260928 13:02:12.720018 28922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:02:12.725150 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.720831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:34691 (local address 127.11.11.126:38221)
0928 13:02:12.720900 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.720903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.721144 (+ 241us) server_negotiation.cc:407] Connection header received
0928 13:02:12.721612 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.721613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.721626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.721681 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.722320 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.722802 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.723774 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.723963 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.724016 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.724907 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.724909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.724909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:12.724913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.724927 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:12.724931 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.724975 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.725058 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.725085 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:02:12.725390 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.720839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38221 (local address 127.11.11.65:34691)
0928 13:02:12.721007 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.721042 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.721525 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.721737 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.721740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.721748 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:12.722249 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.722253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.722884 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.722885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.723689 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.723691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.724113 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.724114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.724237 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.724760 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:12.724761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:12.724799 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:12.724824 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:12.724987 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:12.724991 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:12.724994 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.725048 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.725132 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 13:02:12.725921 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 28922
I20260928 13:02:12.726020 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:12.726115 29050 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38221
I20260928 13:02:12.726234 29050 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:12.726243 11309 external_mini_cluster.cc:1428] Setting key 2ea56c629b67e0ec19e331b5dcf01a1b
I20260928 13:02:12.726450 29050 heartbeater.cc:511] Master 127.11.11.126:38221 requested a full tablet report, sending...
I20260928 13:02:12.727082 28861 ts_manager.cc:194] Registered new tserver with Master: 9bf78fcd5afc4cb2b116a489a54c16d9 (127.11.11.65:41751)
I20260928 13:02:12.727317 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--builtin_ntp_servers=127.11.11.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:02:12.727843 28861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.65:34691
W20260928 13:02:12.809882 29053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:12.810137 29053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:12.810173 29053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:12.811324 29053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:12.811390 29053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:12.811430 29053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:12.812803 29053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:12.813082 29053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:12.813386 29053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:12.818334 29058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.818423 29059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.818487 29053 server_base.cc:1034] running on GCE node
W20260928 13:02:12.818346 29061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.818827 29053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:12.819149 29053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:12.820324 29053 hybrid_clock.cc:648] HybridClock initialized: now 1790600532820294 us; error 44 us; skew 500 ppm
I20260928 13:02:12.821830 29053 webserver.cc:460] Webserver started at http://127.11.11.66:41695/ using document root <none> and password file <none>
I20260928 13:02:12.822084 29053 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:12.822153 29053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:12.822315 29053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:12.823370 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "7cea654bc26046cdb43ed257298a7732"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "9a3062914140170c0019e9aa91ccf1bf"
server_key_iv: "fe342b6a65e12a8420ee88f73d2a7704"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.823751 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "7cea654bc26046cdb43ed257298a7732"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "9a3062914140170c0019e9aa91ccf1bf"
server_key_iv: "fe342b6a65e12a8420ee88f73d2a7704"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.825084 29053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:12.825783 29067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.825956 29053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:12.826050 29053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "7cea654bc26046cdb43ed257298a7732"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "9a3062914140170c0019e9aa91ccf1bf"
server_key_iv: "fe342b6a65e12a8420ee88f73d2a7704"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.826135 29053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:12.841431 29053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:12.841778 29053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:12.841935 29053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:12.842170 29053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:12.842502 29053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:12.842554 29053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.842595 29053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:12.842623 29053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.848874 29053 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:38645
I20260928 13:02:12.848922 29180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:38645 every 8 connection(s)
I20260928 13:02:12.849256 29053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:12.852475 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29053
I20260928 13:02:12.852586 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:12.852799 11309 external_mini_cluster.cc:1428] Setting key b01a48bb6b6a3d262a33c380bbe6db95
I20260928 13:02:12.854014 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--builtin_ntp_servers=127.11.11.84:38429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:02:12.854909 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.850007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:55689 (local address 127.11.11.126:38221)
0928 13:02:12.850060 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.850063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.850330 (+ 267us) server_negotiation.cc:407] Connection header received
0928 13:02:12.850819 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.850820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.850830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.850879 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.851523 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.852008 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.853410 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.853657 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.853737 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.854614 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.854615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.854616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:12.854621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.854639 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:12.854644 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.854704 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.854765 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.854792 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:12.855024 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.850008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38221 (local address 127.11.11.66:55689)
0928 13:02:12.850208 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.850244 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.850730 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.850941 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.850945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.850952 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:12.851457 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.851461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.852117 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.852118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.853333 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.853336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.853750 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.853751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.853881 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.854478 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:12.854478 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:12.854507 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:12.854529 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:12.854704 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:12.854706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:12.854708 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.854756 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.854832 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 13:02:12.855497 29181 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38221
I20260928 13:02:12.855608 29181 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:12.855806 29181 heartbeater.cc:511] Master 127.11.11.126:38221 requested a full tablet report, sending...
I20260928 13:02:12.856208 28861 ts_manager.cc:194] Registered new tserver with Master: 7cea654bc26046cdb43ed257298a7732 (127.11.11.66:38645)
I20260928 13:02:12.856714 28861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.66:55689
W20260928 13:02:12.936911 29184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:12.937130 29184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:12.937153 29184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:12.938386 29184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:12.938433 29184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:12.938459 29184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:12.939831 29184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:38429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:38221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:12.940109 29184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:12.940426 29184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:12.945565 29192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.945690 29184 server_base.cc:1034] running on GCE node
W20260928 13:02:12.945642 29189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:12.945834 29190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:12.946076 29184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:12.946405 29184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:12.947544 29184 hybrid_clock.cc:648] HybridClock initialized: now 1790600532947515 us; error 43 us; skew 500 ppm
I20260928 13:02:12.948984 29184 webserver.cc:460] Webserver started at http://127.11.11.67:39043/ using document root <none> and password file <none>
I20260928 13:02:12.949254 29184 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:12.949325 29184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:12.949474 29184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:12.950548 29184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "f9939470924d48d6ac821ea6eddca565"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "af53139bf67db98828d6592abd055b01"
server_key_iv: "3e7939badeca6381bf3e581cf874fb3f"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.950915 29184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "f9939470924d48d6ac821ea6eddca565"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "af53139bf67db98828d6592abd055b01"
server_key_iv: "3e7939badeca6381bf3e581cf874fb3f"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.952278 29184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:12.953073 29198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.953277 29184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:12.953383 29184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "f9939470924d48d6ac821ea6eddca565"
format_stamp: "Formatted at 2026-09-28 13:02:12 on dist-test-slave-8sjz"
server_key: "af53139bf67db98828d6592abd055b01"
server_key_iv: "3e7939badeca6381bf3e581cf874fb3f"
server_key_version: "encryptionkey@0"
I20260928 13:02:12.953455 29184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:12.962096 29184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:12.962407 29184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:12.962555 29184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:12.962785 29184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:12.963124 29184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:12.963177 29184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.963212 29184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:12.963238 29184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:12.969048 29184 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:46417
I20260928 13:02:12.969092 29311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:46417 every 8 connection(s)
I20260928 13:02:12.969506 29184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:02:12.975090 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.970308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:33061 (local address 127.11.11.126:38221)
0928 13:02:12.970352 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.970355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.970665 (+ 310us) server_negotiation.cc:407] Connection header received
0928 13:02:12.971202 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.971204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.971217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.971266 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.972018 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.972576 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.973604 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.973813 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.973891 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.974780 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.974781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.974782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:12.974787 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.974803 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:12.974808 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.974877 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.974970 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.975005 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:12.975294 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.970317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:38221 (local address 127.11.11.67:33061)
0928 13:02:12.970521 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.970555 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.971092 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.971361 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.971365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.971373 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:12.971916 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.971921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.972684 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.972685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.973494 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.973496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.973902 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.973903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.974035 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.974613 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:12.974614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:12.974649 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:12.974673 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:12.974878 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:12.974884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:12.974887 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.974972 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.975076 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260928 13:02:12.975833 29312 heartbeater.cc:347] Connected to a master server at 127.11.11.126:38221
I20260928 13:02:12.975919 29312 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:12.976135 29312 heartbeater.cc:511] Master 127.11.11.126:38221 requested a full tablet report, sending...
I20260928 13:02:12.976497 28861 ts_manager.cc:194] Registered new tserver with Master: f9939470924d48d6ac821ea6eddca565 (127.11.11.67:46417)
I20260928 13:02:12.976888 28861 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.67:33061
I20260928 13:02:12.979578 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29184
I20260928 13:02:12.979704 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:12.979908 11309 external_mini_cluster.cc:1428] Setting key 857939b1dc5793a202fc7300972f712b
I20260928 13:02:12.980967 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:12.985360 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.982350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.11.11.126:38221)
0928 13:02:12.982393 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.982396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.982503 (+ 107us) server_negotiation.cc:407] Connection header received
0928 13:02:12.982582 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.982583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.982596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.982653 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:12.982817 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.983307 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.984123 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.984333 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.984406 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.985120 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:12.985121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:12.985122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:12.985126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:12.985142 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:12.985146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:12.985194 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.985244 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.985270 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 13:02:12.986955 28861 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:12.988575 28861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:12.995115 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.990818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:46417 (local address 127.0.0.1:52718)
0928 13:02:12.990992 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.991037 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.991151 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.991614 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.991616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.991618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.991668 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.991670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.992735 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.992737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.993930 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.993933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.994815 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.994894 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.994909 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.994976 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.995026 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260928 13:02:12.995254 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.990998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.11.11.66:38645)
0928 13:02:12.991169 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.991174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.991240 (+ 66us) server_negotiation.cc:407] Connection header received
0928 13:02:12.991333 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.991335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.991354 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.991441 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.991629 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.992888 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994211 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.994910 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994972 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.995052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.995127 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.995162 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 13:02:12.995229 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.990825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:38645 (local address 127.0.0.1:55956)
0928 13:02:12.991185 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.991195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.991255 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.991446 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.991449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.991451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.991527 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.991530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.992988 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.992990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.994124 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.994126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.994985 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.995050 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.995054 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.995112 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.995152 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":42,"thread_start_us":107,"threads_started":1}
I20260928 13:02:12.995703 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.990934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.11.11.67:46417)
0928 13:02:12.991467 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.991471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.991482 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:12.991518 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.991520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.991535 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.991616 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.991765 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.992638 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994008 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.994718 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994797 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.994892 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.995558 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.995603 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":100,"threads_started":1}
I20260928 13:02:12.995900 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.991039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:41751 (local address 127.0.0.1:53750)
0928 13:02:12.991289 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:12.991298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:12.991340 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:12.991814 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:12.991815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:12.991816 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:12.991846 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.991847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.993125 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.993127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.994782 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:12.994784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.995682 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.995683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:12.995747 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.995751 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:12.995795 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 13:02:12.995833 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":66,"threads_started":1}
I20260928 13:02:12.995924 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:12.991244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.11.11.65:41751)
0928 13:02:12.991420 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:12.991425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:12.991435 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:12.991472 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:12.991474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:12.991488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:12.991749 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:12.991960 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.993046 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.994873 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:12.995611 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:12.995682 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:12.995776 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:12.995807 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 13:02:12.995848 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260928 13:02:12.996294 29246 tablet_service.cc:1467] Processing CreateTablet for tablet d01f4c54ce344ba49804359d08941868 (DEFAULT_TABLE table=test-table [id=f8432de2de24404485c62bc2b3ae1f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:12.996312 29115 tablet_service.cc:1467] Processing CreateTablet for tablet d01f4c54ce344ba49804359d08941868 (DEFAULT_TABLE table=test-table [id=f8432de2de24404485c62bc2b3ae1f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:12.996577 29246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f4c54ce344ba49804359d08941868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.996704 29115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f4c54ce344ba49804359d08941868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:12.999126 29326 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Bootstrap starting.
I20260928 13:02:12.999528 29327 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Bootstrap starting.
I20260928 13:02:12.999836 29326 tablet_bootstrap.cc:654] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:12.999972 28984 tablet_service.cc:1467] Processing CreateTablet for tablet d01f4c54ce344ba49804359d08941868 (DEFAULT_TABLE table=test-table [id=f8432de2de24404485c62bc2b3ae1f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:13.000200 29327 tablet_bootstrap.cc:654] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:13.000203 29326 log.cc:824] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:13.000386 28984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01f4c54ce344ba49804359d08941868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:13.000530 29327 log.cc:824] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:13.001006 29326 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: No bootstrap required, opened a new log
I20260928 13:02:13.001066 29326 ts_tablet_manager.cc:1397] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:13.001258 29327 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: No bootstrap required, opened a new log
I20260928 13:02:13.001314 29327 ts_tablet_manager.cc:1397] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:02:13.003535 29327 raft_consensus.cc:348] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.003682 29327 raft_consensus.cc:374] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:13.003732 29327 raft_consensus.cc:729] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cea654bc26046cdb43ed257298a7732, State: Initialized, Role: FOLLOWER
I20260928 13:02:13.003705 29326 raft_consensus.cc:348] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.003858 29326 raft_consensus.cc:374] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:13.003891 29326 raft_consensus.cc:729] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9939470924d48d6ac821ea6eddca565, State: Initialized, Role: FOLLOWER
I20260928 13:02:13.003876 29327 consensus_queue.cc:260] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.004065 29326 consensus_queue.cc:260] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.004154 29327 ts_tablet_manager.cc:1428] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:13.004283 29181 heartbeater.cc:503] Master 127.11.11.126:38221 was elected leader, sending a full tablet report...
I20260928 13:02:13.004314 29326 ts_tablet_manager.cc:1428] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:02:13.004391 29312 heartbeater.cc:503] Master 127.11.11.126:38221 was elected leader, sending a full tablet report...
I20260928 13:02:13.004500 29330 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: Bootstrap starting.
I20260928 13:02:13.005168 29330 tablet_bootstrap.cc:654] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:13.005457 29330 log.cc:824] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:13.006247 29330 tablet_bootstrap.cc:492] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: No bootstrap required, opened a new log
I20260928 13:02:13.006387 29330 ts_tablet_manager.cc:1397] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:13.008812 29330 raft_consensus.cc:348] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.008968 29330 raft_consensus.cc:374] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:13.009016 29330 raft_consensus.cc:729] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf78fcd5afc4cb2b116a489a54c16d9, State: Initialized, Role: FOLLOWER
I20260928 13:02:13.009150 29330 consensus_queue.cc:260] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.009392 29330 ts_tablet_manager.cc:1428] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:02:13.009443 29050 heartbeater.cc:503] Master 127.11.11.126:38221 was elected leader, sending a full tablet report...
I20260928 13:02:13.092557 29334 raft_consensus.cc:482] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:13.092687 29334 raft_consensus.cc:504] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.093132 29334 leader_election.cc:288] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cea654bc26046cdb43ed257298a7732 (127.11.11.66:38645), f9939470924d48d6ac821ea6eddca565 (127.11.11.67:46417)
I20260928 13:02:13.096271 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:13.093567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:40893 (local address 127.11.11.67:46417)
0928 13:02:13.093693 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:13.093695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:13.093707 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:02:13.093739 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:13.093740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:13.093749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:13.093827 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:13.093989 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.094457 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095458 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.095841 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:13.095937 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:13.096146 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 13:02:13.096192 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 13:02:13.096272 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:13.093448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:38157 (local address 127.11.11.66:38645)
0928 13:02:13.093501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:13.093503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:13.093657 (+ 154us) server_negotiation.cc:407] Connection header received
0928 13:02:13.093717 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:13.093718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:13.093732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:13.093793 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:13.093949 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.094424 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095462 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.095841 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:13.095934 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:13.096148 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 13:02:13.096197 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:02:13.096282 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:13.093368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:46417 (local address 127.11.11.65:40893)
0928 13:02:13.093430 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:13.093444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:13.093511 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:13.093832 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:13.093833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:13.093836 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:13.093912 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:13.093914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.094544 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.094545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:13.095362 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:13.095364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.095927 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:13.096014 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:13.096017 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:13.096128 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 13:02:13.096180 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":4}
I20260928 13:02:13.096295 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:13.093383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:38645 (local address 127.11.11.65:38157)
0928 13:02:13.093566 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:13.093575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:13.093650 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:13.093795 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:13.093796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:13.093798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:13.093877 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:13.093879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.094503 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.094504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:13.095365 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:13.095368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.095925 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.095926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:13.096014 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:13.096018 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:13.096142 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 13:02:13.096182 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":50,"thread_start_us":112,"threads_started":1}
I20260928 13:02:13.096688 29135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01f4c54ce344ba49804359d08941868" candidate_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cea654bc26046cdb43ed257298a7732" is_pre_election: true
I20260928 13:02:13.096733 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01f4c54ce344ba49804359d08941868" candidate_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9939470924d48d6ac821ea6eddca565" is_pre_election: true
I20260928 13:02:13.096863 29266 raft_consensus.cc:2454] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf78fcd5afc4cb2b116a489a54c16d9 in term 0.
I20260928 13:02:13.096863 29135 raft_consensus.cc:2454] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bf78fcd5afc4cb2b116a489a54c16d9 in term 0.
I20260928 13:02:13.097108 28940 leader_election.cc:302] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cea654bc26046cdb43ed257298a7732, 9bf78fcd5afc4cb2b116a489a54c16d9; no voters:
I20260928 13:02:13.097256 29334 raft_consensus.cc:2781] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:13.097316 29334 raft_consensus.cc:482] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:13.097347 29334 raft_consensus.cc:3037] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:13.098063 29334 raft_consensus.cc:504] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.098269 29334 leader_election.cc:288] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [CANDIDATE]: Term 1 election: Requested vote from peers 7cea654bc26046cdb43ed257298a7732 (127.11.11.66:38645), f9939470924d48d6ac821ea6eddca565 (127.11.11.67:46417)
I20260928 13:02:13.098366 29135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01f4c54ce344ba49804359d08941868" candidate_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cea654bc26046cdb43ed257298a7732"
I20260928 13:02:13.098433 29266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01f4c54ce344ba49804359d08941868" candidate_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9939470924d48d6ac821ea6eddca565"
I20260928 13:02:13.098440 29135 raft_consensus.cc:3037] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:13.098500 29266 raft_consensus.cc:3037] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:13.099382 29266 raft_consensus.cc:2454] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf78fcd5afc4cb2b116a489a54c16d9 in term 1.
I20260928 13:02:13.099382 29135 raft_consensus.cc:2454] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bf78fcd5afc4cb2b116a489a54c16d9 in term 1.
I20260928 13:02:13.099578 28940 leader_election.cc:302] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cea654bc26046cdb43ed257298a7732, 9bf78fcd5afc4cb2b116a489a54c16d9; no voters:
I20260928 13:02:13.099704 29334 raft_consensus.cc:2781] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 1 FOLLOWER]: Leader election won for term 1
W20260928 13:02:13.099786 29182 tablet.cc:2369] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:13.099875 29334 raft_consensus.cc:686] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [term 1 LEADER]: Becoming Leader. State: Replica: 9bf78fcd5afc4cb2b116a489a54c16d9, State: Running, Role: LEADER
I20260928 13:02:13.100015 29334 consensus_queue.cc:237] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } }
I20260928 13:02:13.101035 28861 catalog_manager.cc:5495] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bf78fcd5afc4cb2b116a489a54c16d9 (127.11.11.65). New cstate: current_term: 1 leader_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bf78fcd5afc4cb2b116a489a54c16d9" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 41751 } health_report { overall_health: HEALTHY } } }
I20260928 13:02:13.110890 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:13.108338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.11.11.65:41751)
0928 13:02:13.108400 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:13.108402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:13.108436 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:02:13.108529 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:13.108530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:13.108554 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:13.108632 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:13.108768 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.109265 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.110039 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:13.110242 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:13.110319 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:13.110494 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:13.110594 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:13.110696 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:13.110760 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 13:02:13.110781 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:02:13.112726 29135 raft_consensus.cc:1261] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 1 FOLLOWER]: Refusing update from remote peer 9bf78fcd5afc4cb2b116a489a54c16d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:13.112742 29266 raft_consensus.cc:1261] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 1 FOLLOWER]: Refusing update from remote peer 9bf78fcd5afc4cb2b116a489a54c16d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:13.113096 29334 consensus_queue.cc:1035] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9939470924d48d6ac821ea6eddca565" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:13.113216 29336 consensus_queue.cc:1035] T d01f4c54ce344ba49804359d08941868 P 9bf78fcd5afc4cb2b116a489a54c16d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cea654bc26046cdb43ed257298a7732" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:13.115924 29339 mvcc.cc:205] Tried to move back new op lower bound from 7334299783626985472 to 7334299783578492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334299783626985472}]
I20260928 13:02:13.115975 29340 mvcc.cc:205] Tried to move back new op lower bound from 7334299783626985472 to 7334299783578492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334299783626985472}]
I20260928 13:02:13.116339 29341 mvcc.cc:205] Tried to move back new op lower bound from 7334299783626985472 to 7334299783578492928. Current Snapshot: MvccSnapshot[applied={T|T < 7334299783626985472}]
I20260928 13:02:13.119009 28862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:13.119103 28862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:13.120080 28862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d14167e8a7485bbe42e5bb4b34da12: Sending DeleteTablet for 3 replicas of tablet d01f4c54ce344ba49804359d08941868
I20260928 13:02:13.120429 29115 tablet_service.cc:1514] Processing DeleteTablet for tablet d01f4c54ce344ba49804359d08941868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:13 UTC) from {username='slave'} at 127.0.0.1:55956
I20260928 13:02:13.120469 29246 tablet_service.cc:1514] Processing DeleteTablet for tablet d01f4c54ce344ba49804359d08941868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:13 UTC) from {username='slave'} at 127.0.0.1:52718
I20260928 13:02:13.120689 29349 tablet_replica.cc:330] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: stopping tablet replica
I20260928 13:02:13.120716 29350 tablet_replica.cc:330] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: stopping tablet replica
I20260928 13:02:13.120708 28984 tablet_service.cc:1514] Processing DeleteTablet for tablet d01f4c54ce344ba49804359d08941868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:13 UTC) from {username='slave'} at 127.0.0.1:53750
I20260928 13:02:13.120826 29349 raft_consensus.cc:2229] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:13.120857 29350 raft_consensus.cc:2229] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:13.120965 29349 raft_consensus.cc:2258] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:13.120980 29350 raft_consensus.cc:2258] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:13.121371 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28922
I20260928 13:02:13.121435 29350 ts_tablet_manager.cc:1905] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:13.123023 29349 ts_tablet_manager.cc:1905] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:13.123768 29350 ts_tablet_manager.cc:1918] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:13.123849 29350 log.cc:1196] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/d01f4c54ce344ba49804359d08941868
I20260928 13:02:13.124130 29350 ts_tablet_manager.cc:1939] T d01f4c54ce344ba49804359d08941868 P f9939470924d48d6ac821ea6eddca565: Deleting consensus metadata
I20260928 13:02:13.124608 28846 catalog_manager.cc:4841] TS f9939470924d48d6ac821ea6eddca565 (127.11.11.67:46417): tablet d01f4c54ce344ba49804359d08941868 (table test-table [id=f8432de2de24404485c62bc2b3ae1f4b]) successfully deleted
I20260928 13:02:13.124918 29349 ts_tablet_manager.cc:1918] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:13.124976 29349 log.cc:1196] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/d01f4c54ce344ba49804359d08941868
I20260928 13:02:13.125226 29349 ts_tablet_manager.cc:1939] T d01f4c54ce344ba49804359d08941868 P 7cea654bc26046cdb43ed257298a7732: Deleting consensus metadata
I20260928 13:02:13.125612 28849 catalog_manager.cc:4841] TS 7cea654bc26046cdb43ed257298a7732 (127.11.11.66:38645): tablet d01f4c54ce344ba49804359d08941868 (table test-table [id=f8432de2de24404485c62bc2b3ae1f4b]) successfully deleted
W20260928 13:02:13.126986 28849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:41751 (error 108)
W20260928 13:02:13.127504 29324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:13.127384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:41751 (local address 127.0.0.1:53774)
0928 13:02:13.127436 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:13.127473 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:41751: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:02:13.127669 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29053
W20260928 13:02:13.127676 28849 catalog_manager.cc:4584] TS 9bf78fcd5afc4cb2b116a489a54c16d9 (127.11.11.65:41751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d01f4c54ce344ba49804359d08941868: Network error: Client connection negotiation failed: client connection to 127.11.11.65:41751: connect: Connection reset by peer (error 104)
I20260928 13:02:13.133045 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29184
I20260928 13:02:13.138170 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 28831
2026-09-28T13:02:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (766 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:13 dist-test-slave-8sjz krb5kdc[29355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:13 dist-test-slave-8sjz krb5kdc[29355](info): set up 2 sockets
Sep 28 13:02:13 dist-test-slave-8sjz krb5kdc[29355](info): commencing operation
krb5kdc: starting...
W20260928 13:02:14.173038 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:14.299208 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45911
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:14.382025 29368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:14.382246 29368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:14.382269 29368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:14.383360 29368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:14.383395 29368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:14.383410 29368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:14.383422 29368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:14.383435 29368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:14.384913 29368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:14.385114 29368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:14.385283 29368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:14.390347 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.390348 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.390441 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:14.390583 29368 server_base.cc:1034] running on GCE node
I20260928 13:02:14.390884 29368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:14.391171 29368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:14.392352 29368 hybrid_clock.cc:648] HybridClock initialized: now 1790600534392335 us; error 45 us; skew 500 ppm
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:14.395371 29368 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:14.395872 29368 webserver.cc:460] Webserver started at http://127.11.11.126:38677/ using document root <none> and password file <none>
I20260928 13:02:14.396291 29368 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:14.396359 29368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:14.396463 29368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:14.397598 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "bbeca63e56f44436b419b802af639952"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "a1109261ce99216731a8d39410c1f0f3"
server_key_iv: "6089f570516a1096545199fcc2a3627b"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.397926 29368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "bbeca63e56f44436b419b802af639952"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "a1109261ce99216731a8d39410c1f0f3"
server_key_iv: "6089f570516a1096545199fcc2a3627b"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.399266 29368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:14.399950 29383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.400128 29368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:14.400242 29368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "bbeca63e56f44436b419b802af639952"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "a1109261ce99216731a8d39410c1f0f3"
server_key_iv: "6089f570516a1096545199fcc2a3627b"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.400326 29368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:14.423314 29368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:14.426016 29368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:14.426211 29368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:14.430202 29368 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45911
I20260928 13:02:14.430245 29435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45911 every 8 connection(s)
I20260928 13:02:14.430634 29368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:14.431583 29436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:14.434022 29436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: Bootstrap starting.
I20260928 13:02:14.434415 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29368
I20260928 13:02:14.434545 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:14.434736 29436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:14.434791 11309 external_mini_cluster.cc:1428] Setting key 8b3ab84be4b30b4d1b82f9be3aebdad9
I20260928 13:02:14.435034 29436 log.cc:824] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:14.435842 29436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: No bootstrap required, opened a new log
I20260928 13:02:14.438256 29436 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } }
I20260928 13:02:14.438442 29436 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:14.438478 29436 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbeca63e56f44436b419b802af639952, State: Initialized, Role: FOLLOWER
I20260928 13:02:14.438618 29436 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } }
I20260928 13:02:14.438699 29436 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:14.438728 29436 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:14.438776 29436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:14.439478 29436 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } }
I20260928 13:02:14.439656 29436 leader_election.cc:302] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbeca63e56f44436b419b802af639952; no voters:
I20260928 13:02:14.439872 29436 leader_election.cc:288] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:14.439895 29441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:14.440012 29441 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [term 1 LEADER]: Becoming Leader. State: Replica: bbeca63e56f44436b419b802af639952, State: Running, Role: LEADER
I20260928 13:02:14.440196 29436 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:14.440191 29441 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } }
I20260928 13:02:14.440691 29443 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbeca63e56f44436b419b802af639952. Latest consensus state: current_term: 1 leader_uuid: "bbeca63e56f44436b419b802af639952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } } }
I20260928 13:02:14.440785 29443 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:14.441023 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbeca63e56f44436b419b802af639952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbeca63e56f44436b419b802af639952" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45911 } } }
I20260928 13:02:14.441365 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:14.441594 29448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:14.442132 29448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:14.444190 29448 catalog_manager.cc:1347] Generated new cluster ID: b74401bfa2b148aa8842ddfbc1dd43c9
I20260928 13:02:14.444250 29448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:14.446465 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.435359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.11.11.126:45911)
0928 13:02:14.435543 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:14.435554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:14.435628 (+ 74us) server_negotiation.cc:407] Connection header received
0928 13:02:14.436188 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:14.436196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:14.436250 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:14.436370 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:14.437216 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.438147 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.439216 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.439677 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.439809 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.442870 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:14.442874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:14.442874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:14.442888 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:14.444875 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.445266 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.445270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.445270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.445363 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.445579 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.445580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.445580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.445828 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:14.445999 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:14.446098 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 13:02:14.446241 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260928 13:02:14.466295 29448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:14.467077 29448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:14.480332 29448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: Generated new TSK 0
I20260928 13:02:14.480599 29448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:02:14.507697 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:14.590148 29462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:14.590390 29462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:14.590416 29462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:14.591658 29462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:14.591706 29462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:14.591734 29462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:14.593277 29462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:14.593544 29462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:14.593789 29462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:14.599159 29470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.599229 29468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.599159 29467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:14.599402 29462 server_base.cc:1034] running on GCE node
I20260928 13:02:14.599684 29462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:14.599947 29462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:14.601096 29462 hybrid_clock.cc:648] HybridClock initialized: now 1790600534601076 us; error 38 us; skew 500 ppm
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:14.603868 29462 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:02:14.604475 29462 webserver.cc:460] Webserver started at http://127.11.11.65:44233/ using document root <none> and password file <none>
I20260928 13:02:14.604715 29462 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:14.604769 29462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:14.604924 29462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:14.606014 29462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "05abcf4229a34af49683f5a1787ddcfa"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "c75019f326e7128af2e8db2d0f12a288"
server_key_iv: "5a4ee5b396d1865cc34ba7aaa2802183"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.606411 29462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "05abcf4229a34af49683f5a1787ddcfa"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "c75019f326e7128af2e8db2d0f12a288"
server_key_iv: "5a4ee5b396d1865cc34ba7aaa2802183"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.607775 29462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:14.608597 29477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.608781 29462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.608855 29462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "05abcf4229a34af49683f5a1787ddcfa"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "c75019f326e7128af2e8db2d0f12a288"
server_key_iv: "5a4ee5b396d1865cc34ba7aaa2802183"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.608947 29462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:14.624734 29462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:14.627244 29462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:14.627439 29462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:14.627734 29462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:14.628082 29462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:14.628118 29462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.628141 29462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:14.628188 29462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.634361 29462 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:40553
I20260928 13:02:14.634501 29590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:40553 every 8 connection(s)
I20260928 13:02:14.634811 29462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:14.643129 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29462
I20260928 13:02:14.643254 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:14.643530 11309 external_mini_cluster.cc:1428] Setting key ed7a33d90ccd38a0d8c2f107253888a2
I20260928 13:02:14.645576 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.635693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:39019 (local address 127.11.11.126:45911)
0928 13:02:14.635743 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:14.635745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:14.635984 (+ 239us) server_negotiation.cc:407] Connection header received
0928 13:02:14.636925 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:14.636927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:14.636939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:14.636986 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:14.638013 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.638533 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.639567 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.639773 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.639863 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.642473 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:14.642480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:14.642481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:14.642488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:14.644136 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.644573 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.644574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.644575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.644641 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.644901 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.644903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.644903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.645029 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:14.645108 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:14.645390 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 13:02:14.645484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:02:14.645835 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.635645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45911 (local address 127.11.11.65:39019)
0928 13:02:14.635844 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:14.635901 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:14.636798 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:14.637061 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:14.637066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:14.637681 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:14.637898 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.637903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.638643 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.638644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.639471 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.639473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.639877 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.639878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.640027 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.640741 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:14.640754 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:14.642352 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:14.644234 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.644243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.644245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.644476 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.644713 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.644714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.644714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.644830 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.645098 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:14.645105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:14.645253 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 13:02:14.645370 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:02:14.645541 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260928 13:02:14.646363 29591 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45911
I20260928 13:02:14.646462 29591 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:14.646682 29591 heartbeater.cc:511] Master 127.11.11.126:45911 requested a full tablet report, sending...
I20260928 13:02:14.647210 29400 ts_manager.cc:194] Registered new tserver with Master: 05abcf4229a34af49683f5a1787ddcfa (127.11.11.65:40553)
I20260928 13:02:14.647832 29400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:39019
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:02:14.671638 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:14.753623 29596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:14.753846 29596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:14.753870 29596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:14.755014 29596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:14.755060 29596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:14.755086 29596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:14.756556 29596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:14.756803 29596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:14.757045 29596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:14.761976 29601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.761983 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.762007 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:14.762279 29596 server_base.cc:1034] running on GCE node
I20260928 13:02:14.762509 29596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:14.762758 29596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:14.763911 29596 hybrid_clock.cc:648] HybridClock initialized: now 1790600534763897 us; error 29 us; skew 500 ppm
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:14.766543 29596 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:02:14.767009 29596 webserver.cc:460] Webserver started at http://127.11.11.66:36079/ using document root <none> and password file <none>
I20260928 13:02:14.767236 29596 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:14.767282 29596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:14.767447 29596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:14.768567 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "9700f3c614714169910bd89bb49b6d2e"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "b966945cbd362d3fb46c8ad08e3f0ac2"
server_key_iv: "3a99d81ca2f35dd60f1edd687e0b7903"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.768935 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "9700f3c614714169910bd89bb49b6d2e"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "b966945cbd362d3fb46c8ad08e3f0ac2"
server_key_iv: "3a99d81ca2f35dd60f1edd687e0b7903"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.770364 29596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:14.771168 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.771370 29596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:14.771445 29596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "9700f3c614714169910bd89bb49b6d2e"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "b966945cbd362d3fb46c8ad08e3f0ac2"
server_key_iv: "3a99d81ca2f35dd60f1edd687e0b7903"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.771538 29596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:14.795588 29596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:14.797959 29596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:14.798133 29596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:14.798399 29596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:14.798743 29596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:14.798801 29596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.798843 29596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:14.798872 29596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.805194 29596 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:34993
I20260928 13:02:14.805234 29724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:34993 every 8 connection(s)
I20260928 13:02:14.805564 29596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:14.806797 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29596
I20260928 13:02:14.806927 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:14.807154 11309 external_mini_cluster.cc:1428] Setting key 934cbe76971c07159e46a0faa41520e8
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
I20260928 13:02:14.816583 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.806313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:42801 (local address 127.11.11.126:45911)
0928 13:02:14.806380 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:14.806382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:14.806586 (+ 204us) server_negotiation.cc:407] Connection header received
0928 13:02:14.807230 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:14.807231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:14.807243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:14.807293 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:14.808471 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.808991 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.810545 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.810783 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.810854 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.813597 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:14.813598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:14.813599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:14.813605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:14.815020 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.815532 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.815533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.815534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.815587 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.815883 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.815883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.815883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.815990 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:14.816110 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:14.816445 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 13:02:14.816514 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:02:14.816859 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.806312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45911 (local address 127.11.11.66:42801)
0928 13:02:14.806463 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:14.806500 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:14.807102 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:14.807382 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:14.807386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:14.808169 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:14.808375 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.808381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.809095 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.809096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.810448 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.810451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.810879 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.810880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.811033 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.811601 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:14.811614 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:14.813490 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:14.815110 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.815117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.815119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.815447 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.815668 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.815670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.815670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.815814 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.816087 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:14.816093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:14.816286 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 13:02:14.816409 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 13:02:14.816595 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 13:02:14.817435 29725 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45911
I20260928 13:02:14.817529 29725 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:14.817749 29725 heartbeater.cc:511] Master 127.11.11.126:45911 requested a full tablet report, sending...
I20260928 13:02:14.818222 29400 ts_manager.cc:194] Registered new tserver with Master: 9700f3c614714169910bd89bb49b6d2e (127.11.11.66:34993)
I20260928 13:02:14.818671 29400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:42801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:02:14.841467 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:14.923512 29730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:14.923729 29730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:14.923753 29730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:14.924938 29730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:14.924983 29730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:14.925010 29730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:14.926473 29730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:14.926733 29730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:14.926967 29730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:14.932300 29736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:14.932332 29735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:14.932446 29730 server_base.cc:1034] running on GCE node
W20260928 13:02:14.932505 29738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:14.932756 29730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:14.933032 29730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:14.934190 29730 hybrid_clock.cc:648] HybridClock initialized: now 1790600534934166 us; error 45 us; skew 500 ppm
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:14.936923 29730 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:02:14.937340 29730 webserver.cc:460] Webserver started at http://127.11.11.67:38877/ using document root <none> and password file <none>
I20260928 13:02:14.937595 29730 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:14.937665 29730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:14.937824 29730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:14.938916 29730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "39536f919ba1416c8333f68a7425c745"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "efb744dce0a0e57d001378e58d4c1116"
server_key_iv: "1aef22d612f738c0796a54d5dad5c8df"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.939301 29730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "39536f919ba1416c8333f68a7425c745"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "efb744dce0a0e57d001378e58d4c1116"
server_key_iv: "1aef22d612f738c0796a54d5dad5c8df"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.940975 29730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:14.941795 29745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.941996 29730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:02:14.942103 29730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "39536f919ba1416c8333f68a7425c745"
format_stamp: "Formatted at 2026-09-28 13:02:14 on dist-test-slave-8sjz"
server_key: "efb744dce0a0e57d001378e58d4c1116"
server_key_iv: "1aef22d612f738c0796a54d5dad5c8df"
server_key_version: "encryptionkey@0"
I20260928 13:02:14.942183 29730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:14.969672 29730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:14.972384 29730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:14.972550 29730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:14.972834 29730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:14.973173 29730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:14.973234 29730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.973279 29730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:14.973307 29730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:14.979435 29730 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:40713
I20260928 13:02:14.979547 29858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:40713 every 8 connection(s)
I20260928 13:02:14.979874 29730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:02:14 dist-test-slave-8sjz krb5kdc[29355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:14.987237 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29730
I20260928 13:02:14.987344 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:14.987564 11309 external_mini_cluster.cc:1428] Setting key c59d6ef6ca8acf572a3952cfa7663b3c
I20260928 13:02:14.990654 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.980927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:52565 (local address 127.11.11.126:45911)
0928 13:02:14.981006 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:14.981010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:14.981218 (+ 208us) server_negotiation.cc:407] Connection header received
0928 13:02:14.981822 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:14.981824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:14.981838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:14.981889 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:14.983026 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.983610 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.984837 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.985074 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.985168 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.987531 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:14.987534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:14.987534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:14.987544 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:14.989186 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.989634 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.989635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.989636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.989707 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:14.989973 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:14.989974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:14.989974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:14.990106 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:14.990185 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:14.990474 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 13:02:14.990556 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:02:14.990876 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:14.980899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:45911 (local address 127.11.11.67:52565)
0928 13:02:14.981075 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:14.981121 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:14.981692 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:14.981988 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:14.981992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:14.982688 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:14.982911 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.982916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.983739 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.983740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.984733 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:14.984736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:14.985180 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:14.985181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:14.985322 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:14.985832 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:14.985841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:14.987423 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:14.989288 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.989295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.989298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.989542 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.989778 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:14.989778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:14.989779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:14.989902 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:14.990177 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:14.990183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:14.990338 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 13:02:14.990454 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 13:02:14.990626 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 13:02:14.991359 29859 heartbeater.cc:347] Connected to a master server at 127.11.11.126:45911
I20260928 13:02:14.991472 29859 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:14.991631 29859 heartbeater.cc:511] Master 127.11.11.126:45911 requested a full tablet report, sending...
I20260928 13:02:14.992012 29400 ts_manager.cc:194] Registered new tserver with Master: 39536f919ba1416c8333f68a7425c745 (127.11.11.67:40713)
I20260928 13:02:14.992456 29400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:52565
I20260928 13:02:14.999553 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:15.008709 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.001141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.11.11.126:45911)
0928 13:02:15.001211 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.001213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.001310 (+ 97us) server_negotiation.cc:407] Connection header received
0928 13:02:15.001353 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.001354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.001364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.001423 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.001972 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.002520 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.003534 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.003766 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.003855 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.005512 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.005513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.005514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:15.005523 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.007199 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:15.007674 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:15.007675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:15.007676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:15.007742 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:15.008003 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:15.008004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:15.008005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:15.008239 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.008375 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.008518 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.008592 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:02:15.010679 29400 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:15.012671 29400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:15.019596 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:34993 (local address 127.0.0.1:43402)
0928 13:02:15.015388 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.015425 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.015514 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.016129 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.016130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.016132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.016185 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.016186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017593 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.017594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.018442 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.018445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.019354 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.019355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.019415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.019427 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.019484 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.019524 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260928 13:02:15.019646 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.11.11.66:34993)
0928 13:02:15.015937 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.015941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.015951 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:15.015993 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.015995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.016011 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.016121 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.016245 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017487 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.018533 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.019267 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.019346 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.019410 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.019511 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.019553 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260928 13:02:15.020082 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.11.11.65:40553)
0928 13:02:15.015747 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.015752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.015987 (+ 235us) server_negotiation.cc:407] Connection header received
0928 13:02:15.016107 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.016109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.016127 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.016204 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.016376 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017191 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.018815 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.019545 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.019612 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.019665 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.019924 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.019973 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
I20260928 13:02:15.020012 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:40553 (local address 127.0.0.1:32934)
0928 13:02:15.015921 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.015932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.015990 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.016262 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.016262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.016264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.016292 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.016293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017299 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.017300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.018728 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.018730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.019685 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.019686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.019826 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.019831 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.019890 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.019944 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":74,"spinlock_wait_cycles":384,"thread_start_us":99,"threads_started":1}
I20260928 13:02:15.020872 29659 tablet_service.cc:1467] Processing CreateTablet for tablet 25370729dccc4fc0987efe16883a7f67 (DEFAULT_TABLE table=test-table [id=b4ec8fd31177420b8aba9f483704ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.021046 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:40713 (local address 127.0.0.1:53564)
0928 13:02:15.015628 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.015638 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.015678 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.015912 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.015915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.015918 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.016000 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.016003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017429 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.017430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.018723 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.018726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.020809 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.020810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.020882 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.020886 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.020938 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.020980 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":46,"thread_start_us":65,"threads_started":1}
I20260928 13:02:15.021159 29525 tablet_service.cc:1467] Processing CreateTablet for tablet 25370729dccc4fc0987efe16883a7f67 (DEFAULT_TABLE table=test-table [id=b4ec8fd31177420b8aba9f483704ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.021306 29659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25370729dccc4fc0987efe16883a7f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.021337 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.015510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.11.11.67:40713)
0928 13:02:15.015707 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.015712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.015725 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:02:15.015772 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.015775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.015794 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.015888 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.016155 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.017328 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.019716 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.020525 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.020596 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.020648 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.021200 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.021257 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260928 13:02:15.021525 29525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25370729dccc4fc0987efe16883a7f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.021967 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 25370729dccc4fc0987efe16883a7f67 (DEFAULT_TABLE table=test-table [id=b4ec8fd31177420b8aba9f483704ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.022284 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25370729dccc4fc0987efe16883a7f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.024168 29876 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Bootstrap starting.
I20260928 13:02:15.024948 29876 tablet_bootstrap.cc:654] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.025628 29878 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Bootstrap starting.
I20260928 13:02:15.025648 29876 log.cc:824] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.026204 29877 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: Bootstrap starting.
I20260928 13:02:15.026422 29878 tablet_bootstrap.cc:654] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.026557 29876 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: No bootstrap required, opened a new log
I20260928 13:02:15.026648 29876 ts_tablet_manager.cc:1397] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:02:15.027174 29878 log.cc:824] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.027829 29877 tablet_bootstrap.cc:654] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.028486 29877 log.cc:824] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.029408 29878 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: No bootstrap required, opened a new log
I20260928 13:02:15.029479 29878 ts_tablet_manager.cc:1397] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 13:02:15.029381 29876 raft_consensus.cc:348] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.029569 29876 raft_consensus.cc:374] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.029608 29876 raft_consensus.cc:729] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9700f3c614714169910bd89bb49b6d2e, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.029636 29877 tablet_bootstrap.cc:492] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: No bootstrap required, opened a new log
I20260928 13:02:15.029724 29877 ts_tablet_manager.cc:1397] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 13:02:15.029731 29876 consensus_queue.cc:260] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.030061 29876 ts_tablet_manager.cc:1428] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 13:02:15.030187 29725 heartbeater.cc:503] Master 127.11.11.126:45911 was elected leader, sending a full tablet report...
I20260928 13:02:15.031476 29878 raft_consensus.cc:348] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.031663 29878 raft_consensus.cc:374] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.031718 29878 raft_consensus.cc:729] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39536f919ba1416c8333f68a7425c745, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.031840 29878 consensus_queue.cc:260] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.032141 29878 ts_tablet_manager.cc:1428] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:02:15.032624 29859 heartbeater.cc:503] Master 127.11.11.126:45911 was elected leader, sending a full tablet report...
I20260928 13:02:15.033692 29877 raft_consensus.cc:348] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.033955 29877 raft_consensus.cc:374] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.034057 29877 raft_consensus.cc:729] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05abcf4229a34af49683f5a1787ddcfa, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.034232 29877 consensus_queue.cc:260] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.034582 29591 heartbeater.cc:503] Master 127.11.11.126:45911 was elected leader, sending a full tablet report...
I20260928 13:02:15.034829 29877 ts_tablet_manager.cc:1428] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
W20260928 13:02:15.056238 29726 tablet.cc:2369] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:15.057330 29883 raft_consensus.cc:482] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:15.057439 29883 raft_consensus.cc:504] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.057813 29883 leader_election.cc:288] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05abcf4229a34af49683f5a1787ddcfa (127.11.11.65:40553), 9700f3c614714169910bd89bb49b6d2e (127.11.11.66:34993)
I20260928 13:02:15.060806 29885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.058010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:34993 (local address 127.11.11.67:50837)
0928 13:02:15.058170 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.058179 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.058229 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.058388 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.058389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.058391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.058460 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.058462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.059108 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.059109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.059898 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.059900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.060531 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.060609 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.060618 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.060661 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.060698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260928 13:02:15.060811 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.058076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:50837 (local address 127.11.11.66:34993)
0928 13:02:15.058129 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.058132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.058238 (+ 106us) server_negotiation.cc:407] Connection header received
0928 13:02:15.058273 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.058274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.058283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.058350 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.058530 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.059016 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060001 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.060444 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060505 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.060512 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.060687 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.060737 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 13:02:15.060889 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.058056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:35355 (local address 127.11.11.65:40553)
0928 13:02:15.058101 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.058104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.058111 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:02:15.058142 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.058142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.058152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.058234 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.058404 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.058937 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060083 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.060464 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060558 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.060566 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.060776 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.060818 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:02:15.060931 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.057964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:40553 (local address 127.11.11.67:35355)
0928 13:02:15.058003 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.058015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.058062 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.058251 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.058252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.058255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.058329 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.058332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.059021 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.059023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.059903 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.059906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.060570 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.060571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.060692 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.060696 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.060754 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.060793 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":73}
I20260928 13:02:15.061427 29545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25370729dccc4fc0987efe16883a7f67" candidate_uuid: "39536f919ba1416c8333f68a7425c745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05abcf4229a34af49683f5a1787ddcfa" is_pre_election: true
I20260928 13:02:15.061414 29679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25370729dccc4fc0987efe16883a7f67" candidate_uuid: "39536f919ba1416c8333f68a7425c745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9700f3c614714169910bd89bb49b6d2e" is_pre_election: true
I20260928 13:02:15.061621 29545 raft_consensus.cc:2454] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39536f919ba1416c8333f68a7425c745 in term 0.
I20260928 13:02:15.061622 29679 raft_consensus.cc:2454] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39536f919ba1416c8333f68a7425c745 in term 0.
I20260928 13:02:15.061856 29746 leader_election.cc:302] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05abcf4229a34af49683f5a1787ddcfa, 39536f919ba1416c8333f68a7425c745; no voters:
I20260928 13:02:15.061980 29883 raft_consensus.cc:2781] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:15.062036 29883 raft_consensus.cc:482] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:15.062059 29883 raft_consensus.cc:3037] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:15.062748 29883 raft_consensus.cc:504] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.062896 29883 leader_election.cc:288] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [CANDIDATE]: Term 1 election: Requested vote from peers 05abcf4229a34af49683f5a1787ddcfa (127.11.11.65:40553), 9700f3c614714169910bd89bb49b6d2e (127.11.11.66:34993)
I20260928 13:02:15.063026 29545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25370729dccc4fc0987efe16883a7f67" candidate_uuid: "39536f919ba1416c8333f68a7425c745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05abcf4229a34af49683f5a1787ddcfa"
I20260928 13:02:15.063042 29679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25370729dccc4fc0987efe16883a7f67" candidate_uuid: "39536f919ba1416c8333f68a7425c745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9700f3c614714169910bd89bb49b6d2e"
I20260928 13:02:15.063107 29545 raft_consensus.cc:3037] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:15.063112 29679 raft_consensus.cc:3037] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:15.064080 29679 raft_consensus.cc:2454] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39536f919ba1416c8333f68a7425c745 in term 1.
I20260928 13:02:15.064080 29545 raft_consensus.cc:2454] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39536f919ba1416c8333f68a7425c745 in term 1.
I20260928 13:02:15.064258 29746 leader_election.cc:302] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05abcf4229a34af49683f5a1787ddcfa, 39536f919ba1416c8333f68a7425c745; no voters:
I20260928 13:02:15.064368 29883 raft_consensus.cc:2781] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:15.064522 29883 raft_consensus.cc:686] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 1 LEADER]: Becoming Leader. State: Replica: 39536f919ba1416c8333f68a7425c745, State: Running, Role: LEADER
I20260928 13:02:15.064656 29883 consensus_queue.cc:237] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } }
I20260928 13:02:15.065549 29400 catalog_manager.cc:5495] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39536f919ba1416c8333f68a7425c745 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "39536f919ba1416c8333f68a7425c745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39536f919ba1416c8333f68a7425c745" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 40713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 } health_report { overall_health: UNKNOWN } } }
I20260928 13:02:15.080844 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.11.11.67:40713)
0928 13:02:15.076089 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.076092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.076119 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:02:15.076168 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.076169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.076192 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.076277 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:15.076435 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.077348 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.078657 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.078934 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.079032 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.080357 (+ 1325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:15.080507 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:15.080649 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.080712 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.080744 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 13:02:15.083424 29679 raft_consensus.cc:1261] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 1 FOLLOWER]: Refusing update from remote peer 39536f919ba1416c8333f68a7425c745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:15.083846 29886 consensus_queue.cc:1035] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9700f3c614714169910bd89bb49b6d2e" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:15.084084 29545 raft_consensus.cc:1261] T 25370729dccc4fc0987efe16883a7f67 P 05abcf4229a34af49683f5a1787ddcfa [term 1 FOLLOWER]: Refusing update from remote peer 39536f919ba1416c8333f68a7425c745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:15.084410 29886 consensus_queue.cc:1035] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05abcf4229a34af49683f5a1787ddcfa" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:15.087311 29890 mvcc.cc:205] Tried to move back new op lower bound from 7334299791698436096 to 7334299791625650176. Current Snapshot: MvccSnapshot[applied={T|T < 7334299791698436096}]
I20260928 13:02:15.090318 29400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:15.090420 29400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:15.092525 29400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbeca63e56f44436b419b802af639952: Sending DeleteTablet for 3 replicas of tablet 25370729dccc4fc0987efe16883a7f67
I20260928 13:02:15.092922 29793 tablet_service.cc:1514] Processing DeleteTablet for tablet 25370729dccc4fc0987efe16883a7f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:15 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:53564
I20260928 13:02:15.093083 29525 tablet_service.cc:1514] Processing DeleteTablet for tablet 25370729dccc4fc0987efe16883a7f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:15 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:32934
I20260928 13:02:15.093326 29899 tablet_replica.cc:330] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: stopping tablet replica
I20260928 13:02:15.093348 29659 tablet_service.cc:1514] Processing DeleteTablet for tablet 25370729dccc4fc0987efe16883a7f67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:15 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:43402
I20260928 13:02:15.093434 29899 raft_consensus.cc:2229] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:02:15.093575 29899 raft_consensus.cc:2258] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:15.093592 29901 tablet_replica.cc:330] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: stopping tablet replica
I20260928 13:02:15.093804 29901 raft_consensus.cc:2229] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:15.093966 29901 raft_consensus.cc:2258] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:15.093989 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29462
I20260928 13:02:15.095423 29899 ts_tablet_manager.cc:1905] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:15.097720 29899 ts_tablet_manager.cc:1918] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:15.097808 29899 log.cc:1196] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/25370729dccc4fc0987efe16883a7f67
I20260928 13:02:15.098119 29899 ts_tablet_manager.cc:1939] T 25370729dccc4fc0987efe16883a7f67 P 39536f919ba1416c8333f68a7425c745: Deleting consensus metadata
I20260928 13:02:15.098557 29387 catalog_manager.cc:4841] TS 39536f919ba1416c8333f68a7425c745 (127.11.11.67:40713): tablet 25370729dccc4fc0987efe16883a7f67 (table test-table [id=b4ec8fd31177420b8aba9f483704ec0c]) successfully deleted
I20260928 13:02:15.099035 29901 ts_tablet_manager.cc:1905] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:15.100708 29901 ts_tablet_manager.cc:1918] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:15.100790 29901 log.cc:1196] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/25370729dccc4fc0987efe16883a7f67
I20260928 13:02:15.101150 29901 ts_tablet_manager.cc:1939] T 25370729dccc4fc0987efe16883a7f67 P 9700f3c614714169910bd89bb49b6d2e: Deleting consensus metadata
I20260928 13:02:15.101701 29386 catalog_manager.cc:4841] TS 9700f3c614714169910bd89bb49b6d2e (127.11.11.66:34993): tablet 25370729dccc4fc0987efe16883a7f67 (table test-table [id=b4ec8fd31177420b8aba9f483704ec0c]) successfully deleted
W20260928 13:02:15.103094 29384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:40553 (error 108)
I20260928 13:02:15.103783 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.103643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:40553 (local address 127.0.0.1:32938)
0928 13:02:15.103709 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.103746 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 13:02:15.103897 29384 catalog_manager.cc:4584] TS 05abcf4229a34af49683f5a1787ddcfa (127.11.11.65:40553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25370729dccc4fc0987efe16883a7f67: Network error: Client connection negotiation failed: client connection to 127.11.11.65:40553: connect: Connection refused (error 111)
I20260928 13:02:15.103924 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29596
I20260928 13:02:15.111234 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29730
I20260928 13:02:15.117817 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29368
2026-09-28T13:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1983 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T13:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:15Z Disabled control of system clock
I20260928 13:02:15.148698 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43241
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:43241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:15.233054 29906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:15.233376 29906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:15.233399 29906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:15.234596 29906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:15.234649 29906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:15.234663 29906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:15.234678 29906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:15.234689 29906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:15.236002 29906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:43241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:15.236316 29906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:15.236578 29906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:15.241534 29912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.241534 29914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:15.241818 29906 server_base.cc:1034] running on GCE node
W20260928 13:02:15.241565 29911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:15.242172 29906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:15.242463 29906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:15.243664 29906 hybrid_clock.cc:648] HybridClock initialized: now 1790600535243647 us; error 28 us; skew 500 ppm
I20260928 13:02:15.245074 29906 webserver.cc:460] Webserver started at http://127.11.11.126:45325/ using document root <none> and password file <none>
I20260928 13:02:15.245348 29906 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:15.245397 29906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:15.245548 29906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:15.246698 29906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "2c4f917506934527b284895112604060"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "9aaf8e24c6f8771cc52707b871a9ba17"
server_key_iv: "8c3f4e8d63139ab65cc57333fb191aa3"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.247074 29906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "2c4f917506934527b284895112604060"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "9aaf8e24c6f8771cc52707b871a9ba17"
server_key_iv: "8c3f4e8d63139ab65cc57333fb191aa3"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.248484 29906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:15.249266 29920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.249454 29906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:15.249554 29906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "2c4f917506934527b284895112604060"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "9aaf8e24c6f8771cc52707b871a9ba17"
server_key_iv: "8c3f4e8d63139ab65cc57333fb191aa3"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.249646 29906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:15.272634 29906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:15.272985 29906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:15.273149 29906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:15.277259 29906 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:43241
I20260928 13:02:15.277302 29972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:43241 every 8 connection(s)
I20260928 13:02:15.277697 29906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:15.278757 29973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.281245 29973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: Bootstrap starting.
I20260928 13:02:15.281797 29973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.282044 29973 log.cc:824] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.282721 29973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: No bootstrap required, opened a new log
I20260928 13:02:15.283994 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29906
I20260928 13:02:15.284170 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:15.284402 11309 external_mini_cluster.cc:1428] Setting key b085a40eecd25d36ef0d2d925b83903d
I20260928 13:02:15.284579 29973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } }
I20260928 13:02:15.284797 29973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.284847 29973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4f917506934527b284895112604060, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.284996 29973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } }
I20260928 13:02:15.285089 29973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:15.285122 29973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:15.285179 29973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:15.285938 29973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } }
I20260928 13:02:15.286070 29973 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4f917506934527b284895112604060; no voters:
I20260928 13:02:15.286264 29973 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:15.286350 29978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:15.286553 29978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4f917506934527b284895112604060, State: Running, Role: LEADER
I20260928 13:02:15.286587 29973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:15.286708 29978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } }
I20260928 13:02:15.287158 29979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4f917506934527b284895112604060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } } }
I20260928 13:02:15.287256 29979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:15.287425 29980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4f917506934527b284895112604060. Latest consensus state: current_term: 1 leader_uuid: "2c4f917506934527b284895112604060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4f917506934527b284895112604060" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43241 } } }
I20260928 13:02:15.287496 29980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:15.287462 29987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:15.288164 29987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:15.290102 29987 catalog_manager.cc:1347] Generated new cluster ID: 959a40e9dcef48f4ac664559814a9544
I20260928 13:02:15.290156 29987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:15.293171 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.285097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.11.11.126:43241)
0928 13:02:15.285321 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.285331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.285366 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:02:15.285492 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.285499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.285582 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.285762 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.286058 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.287960 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.289061 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.289652 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.289784 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.292330 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.292335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.292336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:15.292573 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.292625 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:15.292645 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.292738 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.292814 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.292913 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 13:02:15.301321 29987 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:15.301937 29987 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:15.309603 29987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: Generated new TSK 0
I20260928 13:02:15.309829 29987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:15.318667 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--builtin_ntp_servers=127.11.11.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:15.403167 29997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:15.403437 29997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:15.403476 29997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:15.404812 29997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:15.404886 29997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:15.404927 29997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:15.406330 29997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:15.406610 29997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:15.406926 29997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:15.412170 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.412210 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.412217 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:15.412535 29997 server_base.cc:1034] running on GCE node
I20260928 13:02:15.412791 29997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:15.413065 29997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:15.414235 29997 hybrid_clock.cc:648] HybridClock initialized: now 1790600535414222 us; error 28 us; skew 500 ppm
I20260928 13:02:15.420029 29997 webserver.cc:460] Webserver started at http://127.11.11.65:40443/ using document root <none> and password file <none>
I20260928 13:02:15.420431 29997 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:15.420490 29997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:15.420799 29997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:15.422039 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "5445c85a577b4a2ca4ed42c1a6bd8974"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "582975caa1cc4292e249c2fbadaa29d6"
server_key_iv: "c672b0fd1924234633b0d5f2252a039d"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.422403 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "5445c85a577b4a2ca4ed42c1a6bd8974"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "582975caa1cc4292e249c2fbadaa29d6"
server_key_iv: "c672b0fd1924234633b0d5f2252a039d"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.423784 29997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:15.424727 30011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.424974 29997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:02:15.425053 29997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "5445c85a577b4a2ca4ed42c1a6bd8974"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "582975caa1cc4292e249c2fbadaa29d6"
server_key_iv: "c672b0fd1924234633b0d5f2252a039d"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.425146 29997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:15.432214 29997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:15.432512 29997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:15.432654 29997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:15.432880 29997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:15.433199 29997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:15.433233 29997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.433259 29997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:15.433271 29997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.439519 29997 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:35091
I20260928 13:02:15.439559 30124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:35091 every 8 connection(s)
I20260928 13:02:15.439939 29997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:02:15.444247 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 29997
I20260928 13:02:15.444355 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:15.444567 11309 external_mini_cluster.cc:1428] Setting key 72035fe08be668b8c863e8d1878003fc
I20260928 13:02:15.445745 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--builtin_ntp_servers=127.11.11.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:02:15.445904 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.440806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:59779 (local address 127.11.11.126:43241)
0928 13:02:15.440866 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.440868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.441087 (+ 219us) server_negotiation.cc:407] Connection header received
0928 13:02:15.441588 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.441589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.441601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.441748 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.442505 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.443021 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.444111 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.444443 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.444522 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.445566 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.445567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.445568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:15.445576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.445594 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:15.445599 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.445663 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.445754 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.445791 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:02:15.446082 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.440787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43241 (local address 127.11.11.65:59779)
0928 13:02:15.440961 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.441005 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.441499 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.441841 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.441847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.441867 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:15.442398 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.442404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.443127 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.443128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.443948 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.443950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.444532 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.444533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.444708 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.445400 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:15.445401 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:15.445444 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:15.445479 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:15.445667 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:15.445672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:15.445675 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.445731 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.445846 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 13:02:15.446614 30125 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43241
I20260928 13:02:15.446756 30125 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:15.446981 30125 heartbeater.cc:511] Master 127.11.11.126:43241 requested a full tablet report, sending...
I20260928 13:02:15.447564 29937 ts_manager.cc:194] Registered new tserver with Master: 5445c85a577b4a2ca4ed42c1a6bd8974 (127.11.11.65:35091)
I20260928 13:02:15.448272 29937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.65:59779
W20260928 13:02:15.529326 30128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:15.529536 30128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:15.529559 30128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:15.530735 30128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:15.530773 30128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:15.530798 30128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:15.532266 30128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:15.532519 30128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:15.532912 30128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:15.538520 30134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.538520 30136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.538520 30133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:15.538805 30128 server_base.cc:1034] running on GCE node
I20260928 13:02:15.539009 30128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:15.539294 30128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:15.540449 30128 hybrid_clock.cc:648] HybridClock initialized: now 1790600535540432 us; error 33 us; skew 500 ppm
I20260928 13:02:15.541906 30128 webserver.cc:460] Webserver started at http://127.11.11.66:37047/ using document root <none> and password file <none>
I20260928 13:02:15.542160 30128 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:15.542212 30128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:15.542346 30128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:15.543464 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "7d7dc3a09867a91765d790cd84a07003"
server_key_iv: "998823b6fd406fe411e40859562ba821"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.543859 30128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "7d7dc3a09867a91765d790cd84a07003"
server_key_iv: "998823b6fd406fe411e40859562ba821"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.545289 30128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:15.546106 30142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.546294 30128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:15.546397 30128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "7d7dc3a09867a91765d790cd84a07003"
server_key_iv: "998823b6fd406fe411e40859562ba821"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.546473 30128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:15.568697 30128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:15.569074 30128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:15.569239 30128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:15.569499 30128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:15.569852 30128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:15.569908 30128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.569952 30128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:15.569975 30128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.576223 30128 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:44907
I20260928 13:02:15.576274 30255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:44907 every 8 connection(s)
I20260928 13:02:15.576617 30128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:15.581867 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30128
I20260928 13:02:15.581987 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:15.582262 11309 external_mini_cluster.cc:1428] Setting key 5757e98ab24d833d4ffdbae7ae8a5a29
I20260928 13:02:15.582398 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.577465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:53869 (local address 127.11.11.126:43241)
0928 13:02:15.577546 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.577549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.577857 (+ 308us) server_negotiation.cc:407] Connection header received
0928 13:02:15.578444 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.578445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.578456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.578500 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.579222 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.579713 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.580842 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.581028 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.581084 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.582030 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.582032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.582033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:15.582038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.582056 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:15.582061 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.582115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.582265 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.582289 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:02:15.582662 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.577414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43241 (local address 127.11.11.66:53869)
0928 13:02:15.577667 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.577750 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.578333 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.578564 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.578567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.578576 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:15.579145 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.579151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.579819 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.579820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.580759 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.580761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.581115 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.581116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.581254 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.581845 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:15.581846 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:15.581890 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:15.581922 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:15.582162 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:15.582169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:15.582172 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.582263 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.582388 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
I20260928 13:02:15.583217 30256 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43241
I20260928 13:02:15.583320 30256 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:15.583412 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--builtin_ntp_servers=127.11.11.84:41013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:02:15.583590 30256 heartbeater.cc:511] Master 127.11.11.126:43241 requested a full tablet report, sending...
I20260928 13:02:15.584009 29937 ts_manager.cc:194] Registered new tserver with Master: fdd3a29822ca44e48a6e9b3b2d4b72f5 (127.11.11.66:44907)
I20260928 13:02:15.584474 29937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.66:53869
W20260928 13:02:15.666824 30259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:15.667093 30259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:15.667129 30259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:15.668336 30259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:15.668409 30259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:15.668452 30259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:15.669857 30259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:43241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:15.670149 30259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:15.670462 30259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:15.676095 30267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.676041 30264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:15.676252 30265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:15.676311 30259 server_base.cc:1034] running on GCE node
I20260928 13:02:15.676529 30259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:15.676831 30259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:15.678009 30259 hybrid_clock.cc:648] HybridClock initialized: now 1790600535677993 us; error 45 us; skew 500 ppm
I20260928 13:02:15.679399 30259 webserver.cc:460] Webserver started at http://127.11.11.67:39771/ using document root <none> and password file <none>
I20260928 13:02:15.679675 30259 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:15.679774 30259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:15.679917 30259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:15.681133 30259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "408a169092fe4baba5aa03da44cef2a7"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "eb49ccaf15bd029afa5ef7df6ad1215f"
server_key_iv: "f187e7875a34a75aba8234efee4ed5dd"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.681522 30259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "408a169092fe4baba5aa03da44cef2a7"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "eb49ccaf15bd029afa5ef7df6ad1215f"
server_key_iv: "f187e7875a34a75aba8234efee4ed5dd"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.682955 30259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:15.683815 30273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.684029 30259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:15.684180 30259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "408a169092fe4baba5aa03da44cef2a7"
format_stamp: "Formatted at 2026-09-28 13:02:15 on dist-test-slave-8sjz"
server_key: "eb49ccaf15bd029afa5ef7df6ad1215f"
server_key_iv: "f187e7875a34a75aba8234efee4ed5dd"
server_key_version: "encryptionkey@0"
I20260928 13:02:15.684271 30259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:15.699395 30259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:15.699759 30259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:15.699920 30259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:15.700227 30259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:15.700603 30259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:15.700663 30259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.700713 30259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:15.700740 30259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:15.706748 30259 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:38235
I20260928 13:02:15.706797 30386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:38235 every 8 connection(s)
I20260928 13:02:15.707170 30259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:02:15.708920 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30259
I20260928 13:02:15.709076 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:15.709290 11309 external_mini_cluster.cc:1428] Setting key c163e6853f9728b0d074ddf540fb0b75
I20260928 13:02:15.713021 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.708062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51161 (local address 127.11.11.126:43241)
0928 13:02:15.708210 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.708212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.708278 (+ 66us) server_negotiation.cc:407] Connection header received
0928 13:02:15.708951 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.708952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.708964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.709073 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.709991 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.710635 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.711644 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.711827 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.711876 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.712777 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.712778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.712778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:15.712783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.712797 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:15.712801 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.712844 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.712932 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.712957 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 13:02:15.713251 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.707951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:43241 (local address 127.11.11.67:51161)
0928 13:02:15.708158 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.708195 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.708727 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.709177 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.709182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.709199 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:15.709879 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.709900 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.710725 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.710726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.711571 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.711573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.711898 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.711899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.712010 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.712641 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:15.712642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:15.712674 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:15.712697 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:15.712858 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:15.712862 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:15.712865 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.712926 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.713012 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
I20260928 13:02:15.713779 30387 heartbeater.cc:347] Connected to a master server at 127.11.11.126:43241
I20260928 13:02:15.713893 30387 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:15.714082 30387 heartbeater.cc:511] Master 127.11.11.126:43241 requested a full tablet report, sending...
I20260928 13:02:15.714594 29937 ts_manager.cc:194] Registered new tserver with Master: 408a169092fe4baba5aa03da44cef2a7 (127.11.11.67:38235)
I20260928 13:02:15.715016 29937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.67:51161
I20260928 13:02:15.721293 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:15.725956 29977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.722746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.11.11.126:43241)
0928 13:02:15.722802 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.722805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.722894 (+ 89us) server_negotiation.cc:407] Connection header received
0928 13:02:15.722990 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.722991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.723005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.723053 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:15.723251 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.723751 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.724604 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.724839 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.724914 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.725602 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:15.725603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:15.725603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:15.725608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:15.725623 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:15.725627 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:15.725744 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.725852 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.725864 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:02:15.727524 29937 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:15.729259 29937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:15.736865 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.731706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:38235 (local address 127.0.0.1:41118)
0928 13:02:15.732137 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.732187 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.732267 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.732584 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.732587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.732590 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.732810 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.732812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.734414 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.734415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.735566 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.735569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.736506 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.736507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.736603 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.736617 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.736703 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.736763 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":20,"thread_start_us":94,"threads_started":1}
I20260928 13:02:15.736980 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.732003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.11.11.66:44907)
0928 13:02:15.732381 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.732386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.732396 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:15.732437 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.732439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.732457 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.732533 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.732785 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.733973 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.735659 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.736624 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.736709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.736804 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.736833 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.736877 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":147,"threads_started":1}
I20260928 13:02:15.737332 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.731706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:44907 (local address 127.0.0.1:54320)
0928 13:02:15.732142 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.732213 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.732285 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.732601 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.732602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.732603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.732710 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.732713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.734090 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.734091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.735566 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.735569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.736721 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.736722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.736774 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.736777 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.736820 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.736853 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
I20260928 13:02:15.737558 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.732007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.11.11.67:38235)
0928 13:02:15.732229 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.732244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.732256 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:02:15.732476 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.732479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.732496 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.732570 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.732888 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.733978 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.735714 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.736389 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.736504 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.736615 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.736919 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.737471 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260928 13:02:15.738174 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.731969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:35091 (local address 127.0.0.1:37600)
0928 13:02:15.732711 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:15.732721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:15.732773 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:15.734257 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:15.734258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:15.734261 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:15.734300 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.734302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.735500 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.735501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.736658 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:15.736661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.737923 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.737924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:15.737992 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.737995 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:15.738064 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:02:15.738107 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":91,"threads_started":1}
I20260928 13:02:15.738197 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:15.732338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.11.11.65:35091)
0928 13:02:15.732719 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:15.732723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:15.734085 (+ 1362us) server_negotiation.cc:407] Connection header received
0928 13:02:15.734133 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:15.734138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:15.734159 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:15.734220 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:15.734394 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.735399 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.737092 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:15.737713 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:15.737787 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:15.737877 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:15.738081 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 13:02:15.738121 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260928 13:02:15.738180 30190 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 (DEFAULT_TABLE table=test-table [id=1a1965843af3433b8f46e9d6f8ad58bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.738258 30321 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 (DEFAULT_TABLE table=test-table [id=1a1965843af3433b8f46e9d6f8ad58bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.738569 30190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb8c6e747394bfba2a9a713411c0333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.738569 30321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb8c6e747394bfba2a9a713411c0333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.738868 30059 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 (DEFAULT_TABLE table=test-table [id=1a1965843af3433b8f46e9d6f8ad58bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:15.739130 30059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb8c6e747394bfba2a9a713411c0333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:15.741976 30401 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Bootstrap starting.
I20260928 13:02:15.742533 30402 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Bootstrap starting.
I20260928 13:02:15.742785 30401 tablet_bootstrap.cc:654] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.743162 30401 log.cc:824] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.743204 30402 tablet_bootstrap.cc:654] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.743515 30402 log.cc:824] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.743876 30403 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Bootstrap starting.
I20260928 13:02:15.744102 30401 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: No bootstrap required, opened a new log
I20260928 13:02:15.744181 30401 ts_tablet_manager.cc:1397] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:02:15.744370 30402 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: No bootstrap required, opened a new log
I20260928 13:02:15.744458 30402 ts_tablet_manager.cc:1397] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:15.744567 30403 tablet_bootstrap.cc:654] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:15.744913 30403 log.cc:824] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:15.745761 30403 tablet_bootstrap.cc:492] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: No bootstrap required, opened a new log
I20260928 13:02:15.745837 30403 ts_tablet_manager.cc:1397] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:15.746559 30402 raft_consensus.cc:348] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.746727 30402 raft_consensus.cc:374] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.746770 30402 raft_consensus.cc:729] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd3a29822ca44e48a6e9b3b2d4b72f5, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.746907 30402 consensus_queue.cc:260] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.747205 30402 ts_tablet_manager.cc:1428] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:15.747246 30256 heartbeater.cc:503] Master 127.11.11.126:43241 was elected leader, sending a full tablet report...
I20260928 13:02:15.747119 30401 raft_consensus.cc:348] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.747606 30401 raft_consensus.cc:374] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.747664 30401 raft_consensus.cc:729] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5445c85a577b4a2ca4ed42c1a6bd8974, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.747781 30401 consensus_queue.cc:260] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.748072 30401 ts_tablet_manager.cc:1428] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 13:02:15.748100 30125 heartbeater.cc:503] Master 127.11.11.126:43241 was elected leader, sending a full tablet report...
I20260928 13:02:15.748445 30403 raft_consensus.cc:348] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.748649 30403 raft_consensus.cc:374] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:15.748698 30403 raft_consensus.cc:729] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408a169092fe4baba5aa03da44cef2a7, State: Initialized, Role: FOLLOWER
I20260928 13:02:15.748811 30403 consensus_queue.cc:260] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:15.749054 30403 ts_tablet_manager.cc:1428] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:02:15.749122 30387 heartbeater.cc:503] Master 127.11.11.126:43241 was elected leader, sending a full tablet report...
W20260928 13:02:15.827171 30257 tablet.cc:2369] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:02:15.940789 30126 tablet.cc:2369] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:02:15.957778 30388 tablet.cc:2369] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:16.093345 30408 raft_consensus.cc:482] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:16.093521 30408 raft_consensus.cc:504] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:16.093997 30408 leader_election.cc:288] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdd3a29822ca44e48a6e9b3b2d4b72f5 (127.11.11.66:44907), 408a169092fe4baba5aa03da44cef2a7 (127.11.11.67:38235)
I20260928 13:02:16.097088 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.094311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:43573 (local address 127.11.11.66:44907)
0928 13:02:16.094374 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:16.094376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:16.094540 (+ 164us) server_negotiation.cc:407] Connection header received
0928 13:02:16.094572 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:16.094573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:16.094585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:16.094645 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:16.094839 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.095296 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096358 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.096732 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096789 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:16.096796 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:16.096968 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 13:02:16.097018 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:02:16.097110 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.094215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:44907 (local address 127.11.11.65:43573)
0928 13:02:16.094432 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:16.094444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:16.094511 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:16.094661 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:16.094661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:16.094663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:16.094752 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:16.094754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.095408 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.095409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:16.096268 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:16.096270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.096821 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:16.096897 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:16.096901 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:16.096943 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 13:02:16.096977 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":19,"thread_start_us":68,"threads_started":1}
I20260928 13:02:16.097141 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.094212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:38235 (local address 127.11.11.65:58989)
0928 13:02:16.094352 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:16.094365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:16.094426 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:16.094570 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:16.094571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:16.094573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:16.094654 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:16.094657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.095361 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.095363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:16.096297 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:16.096299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.096858 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:16.096949 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:16.096954 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:02:16.097011 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:02:16.097047 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 13:02:16.097143 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.094378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:58989 (local address 127.11.11.67:38235)
0928 13:02:16.094424 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:16.094425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:16.094436 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:16.094479 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:16.094480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:16.094490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:16.094551 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:16.094769 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.095273 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096402 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.096749 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.096840 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:16.096848 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:16.097032 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 13:02:16.097070 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:02:16.097460 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb8c6e747394bfba2a9a713411c0333" candidate_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" is_pre_election: true
I20260928 13:02:16.097532 30341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb8c6e747394bfba2a9a713411c0333" candidate_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408a169092fe4baba5aa03da44cef2a7" is_pre_election: true
I20260928 13:02:16.097647 30210 raft_consensus.cc:2454] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5445c85a577b4a2ca4ed42c1a6bd8974 in term 0.
I20260928 13:02:16.097709 30341 raft_consensus.cc:2454] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5445c85a577b4a2ca4ed42c1a6bd8974 in term 0.
I20260928 13:02:16.097972 30013 leader_election.cc:302] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 408a169092fe4baba5aa03da44cef2a7, 5445c85a577b4a2ca4ed42c1a6bd8974, fdd3a29822ca44e48a6e9b3b2d4b72f5; no voters:
I20260928 13:02:16.098120 30408 raft_consensus.cc:2781] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:16.098177 30408 raft_consensus.cc:482] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:16.098210 30408 raft_consensus.cc:3037] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:16.098882 30408 raft_consensus.cc:504] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:16.099032 30408 leader_election.cc:288] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [CANDIDATE]: Term 1 election: Requested vote from peers fdd3a29822ca44e48a6e9b3b2d4b72f5 (127.11.11.66:44907), 408a169092fe4baba5aa03da44cef2a7 (127.11.11.67:38235)
I20260928 13:02:16.099184 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb8c6e747394bfba2a9a713411c0333" candidate_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5"
I20260928 13:02:16.099200 30341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb8c6e747394bfba2a9a713411c0333" candidate_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408a169092fe4baba5aa03da44cef2a7"
I20260928 13:02:16.099283 30210 raft_consensus.cc:3037] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:16.099278 30341 raft_consensus.cc:3037] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:16.099958 30341 raft_consensus.cc:2454] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5445c85a577b4a2ca4ed42c1a6bd8974 in term 1.
I20260928 13:02:16.100116 30210 raft_consensus.cc:2454] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5445c85a577b4a2ca4ed42c1a6bd8974 in term 1.
I20260928 13:02:16.100152 30013 leader_election.cc:302] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 408a169092fe4baba5aa03da44cef2a7, 5445c85a577b4a2ca4ed42c1a6bd8974; no voters:
I20260928 13:02:16.100261 30408 raft_consensus.cc:2781] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:16.100414 30408 raft_consensus.cc:686] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [term 1 LEADER]: Becoming Leader. State: Replica: 5445c85a577b4a2ca4ed42c1a6bd8974, State: Running, Role: LEADER
I20260928 13:02:16.100510 30408 consensus_queue.cc:237] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } }
I20260928 13:02:16.101402 29936 catalog_manager.cc:5495] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5445c85a577b4a2ca4ed42c1a6bd8974 (127.11.11.65). New cstate: current_term: 1 leader_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5445c85a577b4a2ca4ed42c1a6bd8974" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 35091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 } health_report { overall_health: UNKNOWN } } }
I20260928 13:02:16.177258 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.174716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.11.11.65:35091)
0928 13:02:16.174829 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:16.174831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:16.174841 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:16.174882 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:16.174883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:16.174910 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:16.174996 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:16.175172 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.175685 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.176461 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:16.176704 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:16.176762 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:16.176908 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:16.176997 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:16.177068 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:16.177122 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 13:02:16.177133 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 13:02:16.179574 30210 raft_consensus.cc:1261] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 1 FOLLOWER]: Refusing update from remote peer 5445c85a577b4a2ca4ed42c1a6bd8974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:16.179919 30341 raft_consensus.cc:1261] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 1 FOLLOWER]: Refusing update from remote peer 5445c85a577b4a2ca4ed42c1a6bd8974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:16.179947 30408 consensus_queue.cc:1035] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd3a29822ca44e48a6e9b3b2d4b72f5" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:16.180222 30408 consensus_queue.cc:1035] T 3eb8c6e747394bfba2a9a713411c0333 P 5445c85a577b4a2ca4ed42c1a6bd8974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408a169092fe4baba5aa03da44cef2a7" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:16.182636 30415 mvcc.cc:205] Tried to move back new op lower bound from 7334299796188155904 to 7334299795868295168. Current Snapshot: MvccSnapshot[applied={T|T < 7334299796188155904}]
I20260928 13:02:16.182947 30416 mvcc.cc:205] Tried to move back new op lower bound from 7334299796188155904 to 7334299795868295168. Current Snapshot: MvccSnapshot[applied={T|T < 7334299796188155904}]
I20260928 13:02:16.183549 30417 mvcc.cc:205] Tried to move back new op lower bound from 7334299796188155904 to 7334299795868295168. Current Snapshot: MvccSnapshot[applied={T|T < 7334299796188155904}]
I20260928 13:02:16.185384 29936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:16.185495 29936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:16.186336 29936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4f917506934527b284895112604060: Sending DeleteTablet for 3 replicas of tablet 3eb8c6e747394bfba2a9a713411c0333
I20260928 13:02:16.186686 30321 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:16 UTC) from {username='slave'} at 127.0.0.1:41118
I20260928 13:02:16.186733 30059 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:16 UTC) from {username='slave'} at 127.0.0.1:37600
I20260928 13:02:16.186784 30190 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb8c6e747394bfba2a9a713411c0333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:16 UTC) from {username='slave'} at 127.0.0.1:54320
I20260928 13:02:16.187026 30427 tablet_replica.cc:330] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: stopping tablet replica
I20260928 13:02:16.187163 30425 tablet_replica.cc:330] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: stopping tablet replica
I20260928 13:02:16.187284 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29997
I20260928 13:02:16.188596 30425 raft_consensus.cc:2229] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:16.188696 30425 raft_consensus.cc:2258] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:16.188822 30427 raft_consensus.cc:2229] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:16.188973 30427 raft_consensus.cc:2258] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:16.189182 30425 ts_tablet_manager.cc:1905] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:16.189441 30427 ts_tablet_manager.cc:1905] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:16.190896 30425 ts_tablet_manager.cc:1918] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:16.190954 30425 log.cc:1196] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/3eb8c6e747394bfba2a9a713411c0333
I20260928 13:02:16.191133 30425 ts_tablet_manager.cc:1939] T 3eb8c6e747394bfba2a9a713411c0333 P 408a169092fe4baba5aa03da44cef2a7: Deleting consensus metadata
I20260928 13:02:16.191413 30427 ts_tablet_manager.cc:1918] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:16.191478 30427 log.cc:1196] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/3eb8c6e747394bfba2a9a713411c0333
I20260928 13:02:16.191512 29922 catalog_manager.cc:4841] TS 408a169092fe4baba5aa03da44cef2a7 (127.11.11.67:38235): tablet 3eb8c6e747394bfba2a9a713411c0333 (table test-table [id=1a1965843af3433b8f46e9d6f8ad58bc]) successfully deleted
I20260928 13:02:16.191730 30427 ts_tablet_manager.cc:1939] T 3eb8c6e747394bfba2a9a713411c0333 P fdd3a29822ca44e48a6e9b3b2d4b72f5: Deleting consensus metadata
I20260928 13:02:16.192324 29921 catalog_manager.cc:4841] TS fdd3a29822ca44e48a6e9b3b2d4b72f5 (127.11.11.66:44907): tablet 3eb8c6e747394bfba2a9a713411c0333 (table test-table [id=1a1965843af3433b8f46e9d6f8ad58bc]) successfully deleted
W20260928 13:02:16.193689 29921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:35091 (error 108)
I20260928 13:02:16.193919 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30128
I20260928 13:02:16.194592 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:16.194046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:35091 (local address 127.0.0.1:37620)
0928 13:02:16.194517 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:16.194562 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:35091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
W20260928 13:02:16.194890 29921 catalog_manager.cc:4584] TS 5445c85a577b4a2ca4ed42c1a6bd8974 (127.11.11.65:35091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3eb8c6e747394bfba2a9a713411c0333: Network error: Client connection negotiation failed: client connection to 127.11.11.65:35091: connect: Connection refused (error 111)
I20260928 13:02:16.199424 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30259
I20260928 13:02:16.203856 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 29906
2026-09-28T13:02:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:16 dist-test-slave-8sjz krb5kdc[30431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:16 dist-test-slave-8sjz krb5kdc[30431](info): set up 2 sockets
Sep 28 13:02:16 dist-test-slave-8sjz krb5kdc[30431](info): commencing operation
krb5kdc: starting...
W20260928 13:02:17.249141 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:17.375869 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:44381
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:17.458010 30444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:17.458225 30444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:17.458247 30444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:17.459333 30444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:17.459368 30444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:17.459383 30444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:17.459395 30444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:17.459407 30444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:17.460984 30444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:44381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:17.461187 30444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:17.461359 30444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:17.466290 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.466436 30444 server_base.cc:1034] running on GCE node
W20260928 13:02:17.466285 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.466292 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.466800 30444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:17.467060 30444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:17.468220 30444 hybrid_clock.cc:648] HybridClock initialized: now 1790600537468200 us; error 44 us; skew 500 ppm
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:17.470991 30444 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:17.471501 30444 webserver.cc:460] Webserver started at http://127.11.11.126:40777/ using document root <none> and password file <none>
I20260928 13:02:17.471774 30444 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:17.471827 30444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:17.471990 30444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:17.473224 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "6e62be8c0a724ef4a1be1b64f85fb811"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "73fb18b837832471ac71a3430f2b4211"
server_key_iv: "1b70f4505151cc0ea8635c5299c8d8e4"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.473690 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "6e62be8c0a724ef4a1be1b64f85fb811"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "73fb18b837832471ac71a3430f2b4211"
server_key_iv: "1b70f4505151cc0ea8635c5299c8d8e4"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.475098 30444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:17.475946 30459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.476135 30444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:17.476238 30444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "6e62be8c0a724ef4a1be1b64f85fb811"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "73fb18b837832471ac71a3430f2b4211"
server_key_iv: "1b70f4505151cc0ea8635c5299c8d8e4"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.476336 30444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:17.484531 30444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:17.486653 30444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:17.486781 30444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:17.490742 30444 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:44381
I20260928 13:02:17.490808 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:44381 every 8 connection(s)
I20260928 13:02:17.491163 30444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:17.492174 30512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:17.494542 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: Bootstrap starting.
I20260928 13:02:17.495162 30512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:17.495456 30512 log.cc:824] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:17.496291 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: No bootstrap required, opened a new log
I20260928 13:02:17.497931 30512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } }
I20260928 13:02:17.498095 30512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:17.498172 30512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e62be8c0a724ef4a1be1b64f85fb811, State: Initialized, Role: FOLLOWER
I20260928 13:02:17.498361 30512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } }
I20260928 13:02:17.498463 30512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:17.498508 30512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:17.498557 30512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:17.499089 30512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } }
I20260928 13:02:17.499213 30512 leader_election.cc:302] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e62be8c0a724ef4a1be1b64f85fb811; no voters:
I20260928 13:02:17.499404 30512 leader_election.cc:288] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:17.499455 30515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:17.499666 30512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:17.499677 30515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [term 1 LEADER]: Becoming Leader. State: Replica: 6e62be8c0a724ef4a1be1b64f85fb811, State: Running, Role: LEADER
I20260928 13:02:17.499785 30515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } }
I20260928 13:02:17.500262 30517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e62be8c0a724ef4a1be1b64f85fb811. Latest consensus state: current_term: 1 leader_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } } }
I20260928 13:02:17.500399 30516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e62be8c0a724ef4a1be1b64f85fb811" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 44381 } } }
I20260928 13:02:17.500434 30517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:17.500496 30516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:17.500866 30524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:17.501029 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30444
I20260928 13:02:17.501145 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:17.501381 11309 external_mini_cluster.cc:1428] Setting key 59d132921da90e5b865b89692501683b
I20260928 13:02:17.501395 30524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:17.503449 30524 catalog_manager.cc:1347] Generated new cluster ID: bf9c0baefd3d4205bc6350b9e9181b07
I20260928 13:02:17.503506 30524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:17.513165 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:17.502082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.11.11.126:44381)
0928 13:02:17.502234 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:17.502245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:17.502277 (+ 32us) server_negotiation.cc:407] Connection header received
0928 13:02:17.502391 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:17.502398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:17.502451 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:17.502591 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:17.503419 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.504870 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.505894 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.506487 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.506575 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:17.509522 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:17.509526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:17.509526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:17.509539 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:17.511469 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.511888 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.511891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.511892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.511991 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.512238 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.512238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.512239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.512532 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:17.512661 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:17.512799 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 13:02:17.512921 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 13:02:17.525508 30524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:17.526265 30524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:17.534123 30524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: Generated new TSK 0
I20260928 13:02:17.534340 30524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.11.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
Entry for principal kudu/127.11.11.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab.
I20260928 13:02:17.564443 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:17.647854 30538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:17.648108 30538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:17.648134 30538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:17.649438 30538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:17.649484 30538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:17.649513 30538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:17.650997 30538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.65.keytab
--principal=kudu/127.11.11.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:17.651266 30538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:17.651511 30538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:17.656641 30544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.656641 30543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.656728 30546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.657054 30538 server_base.cc:1034] running on GCE node
I20260928 13:02:17.657236 30538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:17.657434 30538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:17.658581 30538 hybrid_clock.cc:648] HybridClock initialized: now 1790600537658564 us; error 30 us; skew 500 ppm
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:17.661232 30538 init.cc:377] Logged in from keytab as kudu/127.11.11.65@KRBTEST.COM (short username kudu)
I20260928 13:02:17.661664 30538 webserver.cc:460] Webserver started at http://127.11.11.65:35631/ using document root <none> and password file <none>
I20260928 13:02:17.661895 30538 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:17.661963 30538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:17.662127 30538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:17.663213 30538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "4e51b981cc584513a3c8eb766c62d653"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "81be78390a0e1312096947487a9a6323"
server_key_iv: "5a32276b7157abaefd92a0b37bfd7c51"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.663589 30538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "4e51b981cc584513a3c8eb766c62d653"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "81be78390a0e1312096947487a9a6323"
server_key_iv: "5a32276b7157abaefd92a0b37bfd7c51"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.664976 30538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:17.665695 30553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.665868 30538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:17.665973 30538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "4e51b981cc584513a3c8eb766c62d653"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "81be78390a0e1312096947487a9a6323"
server_key_iv: "5a32276b7157abaefd92a0b37bfd7c51"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.666046 30538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:17.688805 30538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:17.690814 30538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:17.690985 30538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:17.691233 30538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:17.691550 30538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:17.691609 30538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.691646 30538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:17.691696 30538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.697651 30538 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:38005
I20260928 13:02:17.697775 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:38005 every 8 connection(s)
I20260928 13:02:17.698081 30538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:02:17.699792 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30538
I20260928 13:02:17.699884 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:17.700174 11309 external_mini_cluster.cc:1428] Setting key ab9452132024393823436d6250b04909
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.65@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.66@KRBTEST.COM; defaulting to no policy
I20260928 13:02:17.708878 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:17.698926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:47497 (local address 127.11.11.126:44381)
0928 13:02:17.699007 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:17.699009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:17.699328 (+ 319us) server_negotiation.cc:407] Connection header received
0928 13:02:17.699988 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:17.699989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:17.700005 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:17.700097 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:17.701439 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.701953 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.702985 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.703222 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.703303 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:17.705752 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:17.705754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:17.705754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:17.705763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:17.707246 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.707691 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.707693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.707693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.707757 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.708019 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.708020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.708021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.708201 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:17.708283 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:17.708595 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 13:02:17.708777 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 13:02:17.709013 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:17.698840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44381 (local address 127.11.11.65:47497)
0928 13:02:17.699195 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:17.699234 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:17.699792 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:17.700255 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:17.700261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:17.701039 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:17.701340 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:17.701346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.702062 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.702063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:17.702882 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:17.702884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.703310 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.703311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:17.703446 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:17.704110 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:17.704125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:17.705643 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:17.707347 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:17.707355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:17.707357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:17.707596 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:17.707833 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:17.707833 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:17.707834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:17.707950 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:17.708273 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:17.708280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:17.708442 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 13:02:17.708571 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 13:02:17.708777 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 13:02:17.709722 30667 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44381
I20260928 13:02:17.709853 30667 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:17.710057 30667 heartbeater.cc:511] Master 127.11.11.126:44381 requested a full tablet report, sending...
I20260928 13:02:17.710680 30476 ts_manager.cc:194] Registered new tserver with Master: 4e51b981cc584513a3c8eb766c62d653 (127.11.11.65:38005)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.66@KRBTEST.COM" created.
I20260928 13:02:17.711433 30476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.65@KRBTEST.COM'} at 127.11.11.65:47497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
Entry for principal kudu/127.11.11.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab.
I20260928 13:02:17.727815 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:17.809206 30672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:17.809463 30672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:17.809501 30672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:17.810668 30672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:17.810737 30672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:17.810783 30672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:17.812316 30672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.66.keytab
--principal=kudu/127.11.11.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:17.812589 30672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:17.812831 30672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:17.817894 30678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.818025 30677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.818085 30672 server_base.cc:1034] running on GCE node
W20260928 13:02:17.818185 30680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.818382 30672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:17.818588 30672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:17.819762 30672 hybrid_clock.cc:648] HybridClock initialized: now 1790600537819749 us; error 29 us; skew 500 ppm
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:17.822562 30672 init.cc:377] Logged in from keytab as kudu/127.11.11.66@KRBTEST.COM (short username kudu)
I20260928 13:02:17.823047 30672 webserver.cc:460] Webserver started at http://127.11.11.66:46347/ using document root <none> and password file <none>
I20260928 13:02:17.823258 30672 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:17.823300 30672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:17.823427 30672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:17.824565 30672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "f9c565bf70a6469ca9dae91102bee37a"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "24081e638250e8c8f8099adc312796d0"
server_key_iv: "34f0aaec0d02d1f5f53ae42cc0fd45e5"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.824898 30672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "f9c565bf70a6469ca9dae91102bee37a"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "24081e638250e8c8f8099adc312796d0"
server_key_iv: "34f0aaec0d02d1f5f53ae42cc0fd45e5"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.826130 30672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:17.826841 30687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.827011 30672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:17.827113 30672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "f9c565bf70a6469ca9dae91102bee37a"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "24081e638250e8c8f8099adc312796d0"
server_key_iv: "34f0aaec0d02d1f5f53ae42cc0fd45e5"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.827191 30672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:17.838721 30672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:17.840806 30672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:17.840970 30672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:17.841205 30672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:17.841532 30672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:17.841588 30672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.841630 30672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:17.841656 30672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.848495 30672 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:38511
I20260928 13:02:17.848691 30800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:38511 every 8 connection(s)
I20260928 13:02:17.848941 30672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:17.852726 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30672
I20260928 13:02:17.852850 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:17.853056 11309 external_mini_cluster.cc:1428] Setting key 0e223449a87ac2e2d223b0f61b0dbcfa
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.66@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:17.859503 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:17.849679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:42971 (local address 127.11.11.126:44381)
0928 13:02:17.849757 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:17.849759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:17.849944 (+ 185us) server_negotiation.cc:407] Connection header received
0928 13:02:17.850464 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:17.850465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:17.850475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:17.850526 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:17.851459 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.851979 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.853210 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.853433 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.853490 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:17.855964 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:17.855965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:17.855966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:17.855975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:17.857823 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.858258 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.858260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.858260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.858324 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:17.858675 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:17.858676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:17.858677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:17.858846 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:17.858927 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:17.859248 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 13:02:17.859357 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:02:17.859565 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:17.849632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44381 (local address 127.11.11.66:42971)
0928 13:02:17.849798 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:17.849835 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:17.850374 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:17.850582 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:17.850585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:17.851179 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:17.851380 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:17.851385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.852136 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.852137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:17.853111 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:17.853114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:17.853524 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:17.853525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:17.853676 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:17.854322 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:17.854331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:17.855840 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:17.857919 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:17.857927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:17.857930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:17.858171 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:17.858422 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:17.858423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:17.858424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:17.858571 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:17.858937 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:17.858944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:17.859091 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 13:02:17.859202 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 13:02:17.859355 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 13:02:17.860598 30801 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44381
I20260928 13:02:17.860730 30801 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:17.860953 30801 heartbeater.cc:511] Master 127.11.11.126:44381 requested a full tablet report, sending...
I20260928 13:02:17.861406 30476 ts_manager.cc:194] Registered new tserver with Master: f9c565bf70a6469ca9dae91102bee37a (127.11.11.66:38511)
I20260928 13:02:17.861828 30476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.66@KRBTEST.COM'} at 127.11.11.66:42971
WARNING: no policy specified for kudu/127.11.11.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
Entry for principal kudu/127.11.11.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab.
I20260928 13:02:17.883373 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:17.965039 30806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:17.965317 30806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:17.965358 30806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:17.966570 30806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:17.966661 30806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:17.966706 30806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:17.968238 30806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.67.keytab
--principal=kudu/127.11.11.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:44381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:17.968523 30806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:17.968762 30806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:17.973994 30811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.973994 30814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:17.974006 30812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:17.974577 30806 server_base.cc:1034] running on GCE node
I20260928 13:02:17.974798 30806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:17.975020 30806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:17.976158 30806 hybrid_clock.cc:648] HybridClock initialized: now 1790600537976154 us; error 41 us; skew 500 ppm
Sep 28 13:02:17 dist-test-slave-8sjz krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:17.978849 30806 init.cc:377] Logged in from keytab as kudu/127.11.11.67@KRBTEST.COM (short username kudu)
I20260928 13:02:17.979347 30806 webserver.cc:460] Webserver started at http://127.11.11.67:39223/ using document root <none> and password file <none>
I20260928 13:02:17.979607 30806 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:17.979660 30806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:17.979823 30806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:17.980983 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "3bb1979583a84aa6914167a1d50abb99"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "735b36b7b536ab7447e55add6bdcb77d"
server_key_iv: "8587dd626bf4d9384ee04f0af2cf35a9"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.981353 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "3bb1979583a84aa6914167a1d50abb99"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "735b36b7b536ab7447e55add6bdcb77d"
server_key_iv: "8587dd626bf4d9384ee04f0af2cf35a9"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.982754 30806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:17.983474 30821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:17.983698 30806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:17.983807 30806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "3bb1979583a84aa6914167a1d50abb99"
format_stamp: "Formatted at 2026-09-28 13:02:17 on dist-test-slave-8sjz"
server_key: "735b36b7b536ab7447e55add6bdcb77d"
server_key_iv: "8587dd626bf4d9384ee04f0af2cf35a9"
server_key_version: "encryptionkey@0"
I20260928 13:02:17.983889 30806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:18.001773 30806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:18.004277 30806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:18.004441 30806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:18.004684 30806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:18.005007 30806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:18.005039 30806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.005081 30806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:18.005107 30806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.011096 30806 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:33491
I20260928 13:02:18.011256 30934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:33491 every 8 connection(s)
I20260928 13:02:18.011502 30806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:02:18 dist-test-slave-8sjz krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600537, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.67@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:18.018622 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30806
I20260928 13:02:18.018728 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:18.018951 11309 external_mini_cluster.cc:1428] Setting key 59711c9d9f1c815e6dcf70f741f69d57
I20260928 13:02:18.022140 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.012274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51531 (local address 127.11.11.126:44381)
0928 13:02:18.012354 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.012357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.012558 (+ 201us) server_negotiation.cc:407] Connection header received
0928 13:02:18.013121 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.013122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.013133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.013179 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.014418 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.014979 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.016027 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.016291 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.016352 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.018618 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.018620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.018620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:18.018628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.020560 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:18.021037 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:18.021039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:18.021039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:18.021108 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:18.021405 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:18.021406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:18.021407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:18.021544 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.021627 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.021939 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.022038 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:02:18.022367 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.012242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:44381 (local address 127.11.11.67:51531)
0928 13:02:18.012426 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.012465 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.013005 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.013280 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.013285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.013989 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:18.014290 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.014298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.015066 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.015066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.015939 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.015941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.016370 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.016371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.016484 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.016943 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:18.016952 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:18.018421 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:18.020681 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:18.020687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:18.020690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:18.020940 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:18.021201 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:18.021202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:18.021202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:18.021330 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:18.021618 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:18.021625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:18.021788 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.021913 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.022096 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260928 13:02:18.022843 30935 heartbeater.cc:347] Connected to a master server at 127.11.11.126:44381
I20260928 13:02:18.022934 30935 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:18.023155 30935 heartbeater.cc:511] Master 127.11.11.126:44381 requested a full tablet report, sending...
I20260928 13:02:18.023594 30476 ts_manager.cc:194] Registered new tserver with Master: 3bb1979583a84aa6914167a1d50abb99 (127.11.11.67:33491)
I20260928 13:02:18.023995 30476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.11.67@KRBTEST.COM'} at 127.11.11.67:51531
I20260928 13:02:18.031081 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:18.039026 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.032496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.11.11.126:44381)
0928 13:02:18.032578 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.032580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.032703 (+ 123us) server_negotiation.cc:407] Connection header received
0928 13:02:18.032790 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.032791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.032809 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.032871 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.033468 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.033963 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.034853 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.035052 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.035156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.036444 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.036445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.036446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:18.036453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.037699 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:18.038103 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:18.038104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:18.038105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:18.038168 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:18.038384 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:18.038385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:18.038385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:18.038581 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.038707 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.038827 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.038914 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:02:18.040824 30476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:18.042418 30476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:18.049842 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:33491 (local address 127.0.0.1:54982)
0928 13:02:18.045613 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.045685 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.045755 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.046247 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.046248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.046251 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.046300 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.046302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047670 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.047672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.048702 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.048704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049472 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.049559 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049575 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.049676 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.049739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260928 13:02:18.049937 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.11.11.65:38005)
0928 13:02:18.045884 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.045889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.045899 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:18.045941 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.046312 (+ 371us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.046343 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.046427 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.046519 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047192 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.048829 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049561 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049647 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049693 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.049801 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.049851 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260928 13:02:18.050130 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.11.11.66:38511)
0928 13:02:18.045645 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.045650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.045658 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:02:18.045697 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.045699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.045712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.045903 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.046152 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047385 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.048829 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049561 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049636 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049693 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.049988 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.050037 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 13:02:18.050261 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.11.11.67:33491)
0928 13:02:18.046083 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.046087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.046096 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:02:18.046146 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.046148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.046165 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.046238 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.046386 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047553 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.048798 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049370 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049469 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049538 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.050108 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.050158 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":80,"threads_started":1}
I20260928 13:02:18.049948 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38005 (local address 127.0.0.1:41130)
0928 13:02:18.045346 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.045394 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.045469 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.046424 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.046425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.046427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.046459 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.046460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047293 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.047294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.048727 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.048729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049646 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.049718 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049722 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.049785 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.049822 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":69,"thread_start_us":72,"threads_started":1}
I20260928 13:02:18.050386 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.045185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:38511 (local address 127.0.0.1:56708)
0928 13:02:18.045356 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.045394 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.045469 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.045909 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.045912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.045915 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.046002 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.046004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.047491 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.047491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.048726 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.048729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.049730 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.049731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.049800 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.049805 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.049855 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.049899 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260928 13:02:18.051082 30601 tablet_service.cc:1467] Processing CreateTablet for tablet 8e322344845147469503d2771b9e94b4 (DEFAULT_TABLE table=test-table [id=48fa973a5a874badbc4bb777e320592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.051095 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 8e322344845147469503d2771b9e94b4 (DEFAULT_TABLE table=test-table [id=48fa973a5a874badbc4bb777e320592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.051095 30869 tablet_service.cc:1467] Processing CreateTablet for tablet 8e322344845147469503d2771b9e94b4 (DEFAULT_TABLE table=test-table [id=48fa973a5a874badbc4bb777e320592a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.051406 30601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e322344845147469503d2771b9e94b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.051421 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e322344845147469503d2771b9e94b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.051421 30869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e322344845147469503d2771b9e94b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.054570 30952 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: Bootstrap starting.
I20260928 13:02:18.055037 30954 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Bootstrap starting.
I20260928 13:02:18.055244 30953 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Bootstrap starting.
I20260928 13:02:18.055457 30952 tablet_bootstrap.cc:654] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.055737 30954 tablet_bootstrap.cc:654] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.055835 30953 tablet_bootstrap.cc:654] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.056128 30952 log.cc:824] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.056392 30954 log.cc:824] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.056507 30953 log.cc:824] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.057029 30952 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: No bootstrap required, opened a new log
I20260928 13:02:18.057114 30952 ts_tablet_manager.cc:1397] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 13:02:18.057183 30954 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: No bootstrap required, opened a new log
I20260928 13:02:18.057189 30953 tablet_bootstrap.cc:492] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: No bootstrap required, opened a new log
I20260928 13:02:18.057260 30953 ts_tablet_manager.cc:1397] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:02:18.057260 30954 ts_tablet_manager.cc:1397] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:02:18.058828 30953 raft_consensus.cc:348] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.058997 30953 raft_consensus.cc:374] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.059041 30953 raft_consensus.cc:729] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb1979583a84aa6914167a1d50abb99, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.059156 30953 consensus_queue.cc:260] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.059448 30953 ts_tablet_manager.cc:1428] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:02:18.059340 30952 raft_consensus.cc:348] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.059383 30954 raft_consensus.cc:348] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.059526 30954 raft_consensus.cc:374] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.059522 30952 raft_consensus.cc:374] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.059571 30954 raft_consensus.cc:729] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c565bf70a6469ca9dae91102bee37a, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.059571 30952 raft_consensus.cc:729] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e51b981cc584513a3c8eb766c62d653, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.059638 30935 heartbeater.cc:503] Master 127.11.11.126:44381 was elected leader, sending a full tablet report...
I20260928 13:02:18.059711 30954 consensus_queue.cc:260] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.059702 30952 consensus_queue.cc:260] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.060002 30952 ts_tablet_manager.cc:1428] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:18.060001 30954 ts_tablet_manager.cc:1428] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:18.060155 30667 heartbeater.cc:503] Master 127.11.11.126:44381 was elected leader, sending a full tablet report...
I20260928 13:02:18.060177 30801 heartbeater.cc:503] Master 127.11.11.126:44381 was elected leader, sending a full tablet report...
I20260928 13:02:18.095180 30958 raft_consensus.cc:482] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:18.095363 30958 raft_consensus.cc:504] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.095881 30958 leader_election.cc:288] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9c565bf70a6469ca9dae91102bee37a (127.11.11.66:38511), 4e51b981cc584513a3c8eb766c62d653 (127.11.11.65:38005)
W20260928 13:02:18.099485 30802 tablet.cc:2369] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:18.099851 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.096120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38005 (local address 127.11.11.67:51985)
0928 13:02:18.096177 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.096191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.096240 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.096423 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.096425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.096426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.096507 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.096513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.097469 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.097470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.098772 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.098776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.099575 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.099576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.099659 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.099663 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.099722 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.099771 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":37}
I20260928 13:02:18.099865 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.096166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:51985 (local address 127.11.11.65:38005)
0928 13:02:18.096228 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.096231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.096242 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:18.096300 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.096301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.096311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.096376 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.096584 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.097390 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.098879 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.099484 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.099567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.099576 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.099743 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.099793 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:02:18.099936 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.096207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:39975 (local address 127.11.11.66:38511)
0928 13:02:18.096262 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.096264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.096572 (+ 308us) server_negotiation.cc:407] Connection header received
0928 13:02:18.096601 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.096603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.096613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.096674 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.096804 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.097484 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.098879 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.099484 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.099567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.099576 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.099821 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.099864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:02:18.100215 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.096180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:38511 (local address 127.11.11.67:39975)
0928 13:02:18.096379 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.096388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.096436 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.096688 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.096689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.096691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.096741 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.096742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.097553 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.097553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.098772 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.098776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.099681 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.099682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.099760 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.099764 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.099814 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.099853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":4,"thread_start_us":115,"threads_started":1}
I20260928 13:02:18.100329 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e322344845147469503d2771b9e94b4" candidate_uuid: "3bb1979583a84aa6914167a1d50abb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e51b981cc584513a3c8eb766c62d653" is_pre_election: true
I20260928 13:02:18.100502 30621 raft_consensus.cc:2454] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb1979583a84aa6914167a1d50abb99 in term 0.
I20260928 13:02:18.100597 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e322344845147469503d2771b9e94b4" candidate_uuid: "3bb1979583a84aa6914167a1d50abb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c565bf70a6469ca9dae91102bee37a" is_pre_election: true
I20260928 13:02:18.100742 30754 raft_consensus.cc:2454] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb1979583a84aa6914167a1d50abb99 in term 0.
I20260928 13:02:18.100739 30824 leader_election.cc:302] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb1979583a84aa6914167a1d50abb99, 4e51b981cc584513a3c8eb766c62d653; no voters:
I20260928 13:02:18.100857 30958 raft_consensus.cc:2781] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:18.100939 30958 raft_consensus.cc:482] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:18.100970 30958 raft_consensus.cc:3037] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.101827 30958 raft_consensus.cc:504] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.101975 30958 leader_election.cc:288] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [CANDIDATE]: Term 1 election: Requested vote from peers f9c565bf70a6469ca9dae91102bee37a (127.11.11.66:38511), 4e51b981cc584513a3c8eb766c62d653 (127.11.11.65:38005)
I20260928 13:02:18.102115 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e322344845147469503d2771b9e94b4" candidate_uuid: "3bb1979583a84aa6914167a1d50abb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c565bf70a6469ca9dae91102bee37a"
I20260928 13:02:18.102188 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e322344845147469503d2771b9e94b4" candidate_uuid: "3bb1979583a84aa6914167a1d50abb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e51b981cc584513a3c8eb766c62d653"
I20260928 13:02:18.102219 30754 raft_consensus.cc:3037] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.102298 30621 raft_consensus.cc:3037] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.102980 30754 raft_consensus.cc:2454] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb1979583a84aa6914167a1d50abb99 in term 1.
I20260928 13:02:18.103039 30621 raft_consensus.cc:2454] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb1979583a84aa6914167a1d50abb99 in term 1.
I20260928 13:02:18.103199 30822 leader_election.cc:302] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb1979583a84aa6914167a1d50abb99, f9c565bf70a6469ca9dae91102bee37a; no voters:
I20260928 13:02:18.103295 30958 raft_consensus.cc:2781] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:18.103466 30958 raft_consensus.cc:686] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb1979583a84aa6914167a1d50abb99, State: Running, Role: LEADER
I20260928 13:02:18.103575 30958 consensus_queue.cc:237] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } }
I20260928 13:02:18.104599 30476 catalog_manager.cc:5495] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb1979583a84aa6914167a1d50abb99 (127.11.11.67). New cstate: current_term: 1 leader_uuid: "3bb1979583a84aa6914167a1d50abb99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb1979583a84aa6914167a1d50abb99" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 } health_report { overall_health: UNKNOWN } } }
I20260928 13:02:18.124969 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.122315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.11.11.67:33491)
0928 13:02:18.122399 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.122400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.122410 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:18.122449 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.122449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.122469 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.122546 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:18.122692 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.123276 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.124082 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.124312 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.124376 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.124549 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:18.124712 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:18.124786 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.124819 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.124839 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:02:18.126991 30754 raft_consensus.cc:1261] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 1 FOLLOWER]: Refusing update from remote peer 3bb1979583a84aa6914167a1d50abb99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:18.127027 30621 raft_consensus.cc:1261] T 8e322344845147469503d2771b9e94b4 P 4e51b981cc584513a3c8eb766c62d653 [term 1 FOLLOWER]: Refusing update from remote peer 3bb1979583a84aa6914167a1d50abb99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:18.127439 30962 consensus_queue.cc:1035] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c565bf70a6469ca9dae91102bee37a" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:18.127570 30958 consensus_queue.cc:1035] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e51b981cc584513a3c8eb766c62d653" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:18.130684 30967 mvcc.cc:205] Tried to move back new op lower bound from 7334299804165312512 to 7334299804073357312. Current Snapshot: MvccSnapshot[applied={T|T < 7334299804165312512}]
I20260928 13:02:18.130750 30965 mvcc.cc:205] Tried to move back new op lower bound from 7334299804165312512 to 7334299804073357312. Current Snapshot: MvccSnapshot[applied={T|T < 7334299804165312512}]
I20260928 13:02:18.133204 30476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:18.133311 30476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:18.134099 30476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e62be8c0a724ef4a1be1b64f85fb811: Sending DeleteTablet for 3 replicas of tablet 8e322344845147469503d2771b9e94b4
I20260928 13:02:18.134462 30601 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e322344845147469503d2771b9e94b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:41130
I20260928 13:02:18.134481 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e322344845147469503d2771b9e94b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:56708
I20260928 13:02:18.134477 30869 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e322344845147469503d2771b9e94b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='kudu', principal='kudu/127.11.11.126@KRBTEST.COM'} at 127.0.0.1:54982
I20260928 13:02:18.134788 30977 tablet_replica.cc:330] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: stopping tablet replica
I20260928 13:02:18.134793 30976 tablet_replica.cc:330] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: stopping tablet replica
I20260928 13:02:18.134896 30976 raft_consensus.cc:2229] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:18.134928 30977 raft_consensus.cc:2229] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:02:18.135011 30976 raft_consensus.cc:2258] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:18.134999 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30538
I20260928 13:02:18.135448 30976 ts_tablet_manager.cc:1905] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:18.135756 30977 raft_consensus.cc:2258] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:18.136260 30977 ts_tablet_manager.cc:1905] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:18.137672 30976 ts_tablet_manager.cc:1918] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:18.137753 30976 log.cc:1196] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/8e322344845147469503d2771b9e94b4
I20260928 13:02:18.137966 30977 ts_tablet_manager.cc:1918] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:18.138037 30976 ts_tablet_manager.cc:1939] T 8e322344845147469503d2771b9e94b4 P f9c565bf70a6469ca9dae91102bee37a: Deleting consensus metadata
I20260928 13:02:18.138137 30977 log.cc:1196] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/8e322344845147469503d2771b9e94b4
I20260928 13:02:18.138468 30977 ts_tablet_manager.cc:1939] T 8e322344845147469503d2771b9e94b4 P 3bb1979583a84aa6914167a1d50abb99: Deleting consensus metadata
I20260928 13:02:18.138440 30460 catalog_manager.cc:4841] TS f9c565bf70a6469ca9dae91102bee37a (127.11.11.66:38511): tablet 8e322344845147469503d2771b9e94b4 (table test-table [id=48fa973a5a874badbc4bb777e320592a]) successfully deleted
I20260928 13:02:18.138890 30463 catalog_manager.cc:4841] TS 3bb1979583a84aa6914167a1d50abb99 (127.11.11.67:33491): tablet 8e322344845147469503d2771b9e94b4 (table test-table [id=48fa973a5a874badbc4bb777e320592a]) successfully deleted
W20260928 13:02:18.140996 30462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:38005 (error 108)
I20260928 13:02:18.141575 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.141429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:38005 (local address 127.0.0.1:41140)
0928 13:02:18.141494 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.141538 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 13:02:18.141637 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30672
W20260928 13:02:18.144158 30462 catalog_manager.cc:4584] TS 4e51b981cc584513a3c8eb766c62d653 (127.11.11.65:38005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e322344845147469503d2771b9e94b4: Network error: Client connection negotiation failed: client connection to 127.11.11.65:38005: connect: Connection refused (error 111)
I20260928 13:02:18.147567 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30806
I20260928 13:02:18.152247 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30444
2026-09-28T13:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1948 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T13:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:18Z Disabled control of system clock
I20260928 13:02:18.180097 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46775
--webserver_interface=127.11.11.126
--webserver_port=0
--builtin_ntp_servers=127.11.11.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:18.261718 30982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:18.261958 30982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:18.261981 30982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:18.263247 30982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:18.263293 30982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:18.263305 30982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:18.263317 30982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:18.263330 30982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:18.264827 30982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:18.265044 30982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:18.265291 30982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:18.270219 30987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.270387 30990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.270394 30988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.270845 30982 server_base.cc:1034] running on GCE node
I20260928 13:02:18.271171 30982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:18.271488 30982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:18.272673 30982 hybrid_clock.cc:648] HybridClock initialized: now 1790600538272660 us; error 33 us; skew 500 ppm
I20260928 13:02:18.273965 30982 webserver.cc:460] Webserver started at http://127.11.11.126:40471/ using document root <none> and password file <none>
I20260928 13:02:18.274245 30982 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:18.274317 30982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:18.274467 30982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:18.275554 30982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "22c577c5b3ea43f5a5af42b347483b9d"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "364b19450b19ca808a2ab63f7897bb9a"
server_key_iv: "d640c95889af3df74dc18ed880ed6eea"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.275925 30982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "22c577c5b3ea43f5a5af42b347483b9d"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "364b19450b19ca808a2ab63f7897bb9a"
server_key_iv: "d640c95889af3df74dc18ed880ed6eea"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.277266 30982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:18.278059 30996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.278252 30982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:18.278359 30982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "22c577c5b3ea43f5a5af42b347483b9d"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "364b19450b19ca808a2ab63f7897bb9a"
server_key_iv: "d640c95889af3df74dc18ed880ed6eea"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.278434 30982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:18.289856 30982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:18.290172 30982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:18.290328 30982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:18.294329 30982 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46775
I20260928 13:02:18.294359 31048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46775 every 8 connection(s)
I20260928 13:02:18.294734 30982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:18.295688 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.298163 31049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: Bootstrap starting.
I20260928 13:02:18.298686 31049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.298925 31049 log.cc:824] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.299556 31049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: No bootstrap required, opened a new log
I20260928 13:02:18.301177 31049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } }
I20260928 13:02:18.301318 31049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.301342 31049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c577c5b3ea43f5a5af42b347483b9d, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.301446 31049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } }
I20260928 13:02:18.301529 31049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:18.301568 31049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:18.301640 31049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.302173 31049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } }
I20260928 13:02:18.302294 31049 leader_election.cc:302] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22c577c5b3ea43f5a5af42b347483b9d; no voters:
I20260928 13:02:18.302474 31049 leader_election.cc:288] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:18.302533 31052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:18.302786 31052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [term 1 LEADER]: Becoming Leader. State: Replica: 22c577c5b3ea43f5a5af42b347483b9d, State: Running, Role: LEADER
I20260928 13:02:18.302809 31049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:18.302894 31052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } }
I20260928 13:02:18.303347 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22c577c5b3ea43f5a5af42b347483b9d. Latest consensus state: current_term: 1 leader_uuid: "22c577c5b3ea43f5a5af42b347483b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } } }
I20260928 13:02:18.303434 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:18.303535 31053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22c577c5b3ea43f5a5af42b347483b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22c577c5b3ea43f5a5af42b347483b9d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46775 } } }
I20260928 13:02:18.303660 31053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:18.303855 31061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:18.304395 31061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:18.304647 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 30982
I20260928 13:02:18.304750 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:18.304937 11309 external_mini_cluster.cc:1428] Setting key 1c61336f2133e0aaa0009c1552bd91b0
I20260928 13:02:18.306020 31061 catalog_manager.cc:1347] Generated new cluster ID: 2ece33ba42f443b187d27ecf607ff956
I20260928 13:02:18.306075 31061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:18.311009 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.305399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.11.11.126:46775)
0928 13:02:18.305583 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.305591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.305624 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:02:18.305740 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.305750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.305819 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.306220 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.306310 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.307676 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.308784 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.309447 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.309580 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.310277 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.310281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.310282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:18.310498 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.310541 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:18.310556 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.310628 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.310706 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.310787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 13:02:18.316711 31061 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:18.317209 31061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:18.320374 31061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: Generated new TSK 0
I20260928 13:02:18.320547 31061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:18.322691 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.65:0
--local_ip_for_outbound_sockets=127.11.11.65
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--builtin_ntp_servers=127.11.11.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:18.404537 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:18.404769 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:18.404807 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:18.405983 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:18.406046 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:18.406091 31073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.65
I20260928 13:02:18.407452 31073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.11.65
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.65
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:18.407709 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:18.407994 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:18.413954 31079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.414119 31073 server_base.cc:1034] running on GCE node
W20260928 13:02:18.413956 31081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.413954 31078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.414438 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:18.414740 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:18.415930 31073 hybrid_clock.cc:648] HybridClock initialized: now 1790600538415912 us; error 36 us; skew 500 ppm
I20260928 13:02:18.417491 31073 webserver.cc:460] Webserver started at http://127.11.11.65:36459/ using document root <none> and password file <none>
I20260928 13:02:18.417749 31073 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:18.417806 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:18.417951 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:18.419363 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data/instance:
uuid: "86809474ffa14feb81ac3721491b98e7"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "617895775fa7352080ae2e3987cb8198"
server_key_iv: "af09b5bf12c1d370a90a38f85536e55c"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.419755 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance:
uuid: "86809474ffa14feb81ac3721491b98e7"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "617895775fa7352080ae2e3987cb8198"
server_key_iv: "af09b5bf12c1d370a90a38f85536e55c"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.421443 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 13:02:18.422376 31087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.422580 31073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:18.422680 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal
uuid: "86809474ffa14feb81ac3721491b98e7"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "617895775fa7352080ae2e3987cb8198"
server_key_iv: "af09b5bf12c1d370a90a38f85536e55c"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.422761 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:18.431370 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:18.431728 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:18.431897 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:18.432161 31073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:18.432529 31073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:18.432587 31073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.432631 31073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:18.432658 31073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.438297 31073 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.65:42179
I20260928 13:02:18.438374 31200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.65:42179 every 8 connection(s)
I20260928 13:02:18.438725 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/data/info.pb
I20260928 13:02:18.444015 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.439478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.65:54283 (local address 127.11.11.126:46775)
0928 13:02:18.439522 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.439523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.439790 (+ 267us) server_negotiation.cc:407] Connection header received
0928 13:02:18.440368 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.440369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.440384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.440439 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.441143 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.441635 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.442682 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.442873 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.442936 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.443775 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.443777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.443777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:18.443781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.443796 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:18.443800 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.443845 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.443931 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.443955 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:18.444290 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.439478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46775 (local address 127.11.11.65:54283)
0928 13:02:18.439648 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.439697 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.440276 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.440493 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.440496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.440504 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:18.441052 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.441057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.441726 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.441727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.442533 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.442536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.442954 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.442955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.443065 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.443634 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:18.443634 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:18.443670 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:18.443696 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:18.443857 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:18.443862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:18.443864 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.443917 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.443996 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 13:02:18.444926 31201 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46775
I20260928 13:02:18.445014 31201 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:18.445168 31201 heartbeater.cc:511] Master 127.11.11.126:46775 requested a full tablet report, sending...
I20260928 13:02:18.445645 31013 ts_manager.cc:194] Registered new tserver with Master: 86809474ffa14feb81ac3721491b98e7 (127.11.11.65:42179)
I20260928 13:02:18.446290 31013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.65:54283
I20260928 13:02:18.447984 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31073
I20260928 13:02:18.448139 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-0/wal/instance
I20260928 13:02:18.448356 11309 external_mini_cluster.cc:1428] Setting key 4b52bf5d758d1f0aaa840413ade1abb2
I20260928 13:02:18.449327 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.66:0
--local_ip_for_outbound_sockets=127.11.11.66
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--builtin_ntp_servers=127.11.11.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:02:18.531667 31204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:18.531886 31204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:18.531909 31204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:18.533099 31204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:18.533140 31204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:18.533167 31204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.66
I20260928 13:02:18.534436 31204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.11.66
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.66
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:18.534619 31204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:18.534869 31204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:18.539904 31209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.539925 31212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.539932 31204 server_base.cc:1034] running on GCE node
W20260928 13:02:18.539914 31210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.540336 31204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:18.540602 31204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:18.541806 31204 hybrid_clock.cc:648] HybridClock initialized: now 1790600538541795 us; error 44 us; skew 500 ppm
I20260928 13:02:18.543221 31204 webserver.cc:460] Webserver started at http://127.11.11.66:37347/ using document root <none> and password file <none>
I20260928 13:02:18.543490 31204 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:18.543565 31204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:18.543737 31204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:18.544884 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data/instance:
uuid: "a956de805d8b49ba9e1e94e9ed16e0fd"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "f06cc808bcf3b76e802d4478c1b50e0c"
server_key_iv: "b67d2b925220560b48bb51ed7ae3cb14"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.545257 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance:
uuid: "a956de805d8b49ba9e1e94e9ed16e0fd"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "f06cc808bcf3b76e802d4478c1b50e0c"
server_key_iv: "b67d2b925220560b48bb51ed7ae3cb14"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.546555 31204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:02:18.547289 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.547487 31204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:02:18.547590 31204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal
uuid: "a956de805d8b49ba9e1e94e9ed16e0fd"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "f06cc808bcf3b76e802d4478c1b50e0c"
server_key_iv: "b67d2b925220560b48bb51ed7ae3cb14"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.547688 31204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:18.554217 31204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:18.554503 31204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:18.554657 31204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:18.554876 31204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:18.555210 31204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:18.555269 31204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.555308 31204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:18.555337 31204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.561292 31204 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.66:36249
I20260928 13:02:18.561333 31331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.66:36249 every 8 connection(s)
I20260928 13:02:18.561671 31204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/data/info.pb
I20260928 13:02:18.564184 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31204
I20260928 13:02:18.564297 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal/instance
I20260928 13:02:18.564503 11309 external_mini_cluster.cc:1428] Setting key da46e22296d99d44aa076e52eb9f2426
I20260928 13:02:18.565482 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.11.67:0
--local_ip_for_outbound_sockets=127.11.11.67
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--builtin_ntp_servers=127.11.11.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:02:18.567602 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.562489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.66:56091 (local address 127.11.11.126:46775)
0928 13:02:18.562584 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.562586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.562770 (+ 184us) server_negotiation.cc:407] Connection header received
0928 13:02:18.563251 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.563252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.563262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.563318 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.564004 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.564776 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.565959 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.566286 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.566361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.567328 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.567329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.567330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:18.567335 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.567353 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:18.567358 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.567422 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.567480 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.567511 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 13:02:18.567857 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.562436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46775 (local address 127.11.11.66:56091)
0928 13:02:18.562629 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.562672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.563162 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.563387 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.563390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.563398 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:18.563895 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.563899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.564888 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.564889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.565876 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.565879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.566407 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.566408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.566522 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.567183 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:18.567183 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:18.567212 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:18.567237 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:18.567427 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:18.567430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:18.567433 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.567481 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.567560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260928 13:02:18.568382 31332 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46775
I20260928 13:02:18.568491 31332 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:18.568672 31332 heartbeater.cc:511] Master 127.11.11.126:46775 requested a full tablet report, sending...
I20260928 13:02:18.569078 31013 ts_manager.cc:194] Registered new tserver with Master: a956de805d8b49ba9e1e94e9ed16e0fd (127.11.11.66:36249)
I20260928 13:02:18.569456 31013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.66:56091
W20260928 13:02:18.648193 31335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:18.648406 31335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:18.648429 31335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:18.649631 31335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:18.649672 31335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:18.649698 31335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.11.67
I20260928 13:02:18.651327 31335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.11.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.11.67
--webserver_port=0
--tserver_master_addrs=127.11.11.126:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.11.67
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:18.651575 31335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:18.651877 31335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:18.656929 31340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.656939 31343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:18.656924 31341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:18.657030 31335 server_base.cc:1034] running on GCE node
I20260928 13:02:18.657352 31335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:18.657621 31335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:18.658795 31335 hybrid_clock.cc:648] HybridClock initialized: now 1790600538658756 us; error 59 us; skew 500 ppm
I20260928 13:02:18.660319 31335 webserver.cc:460] Webserver started at http://127.11.11.67:38419/ using document root <none> and password file <none>
I20260928 13:02:18.660569 31335 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:18.660640 31335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:18.660801 31335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:18.661880 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data/instance:
uuid: "68eef2375f614bc786a25be214145a0f"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "46681ecc52e50a39575bd575b783e361"
server_key_iv: "399f223aeb5a6e6ab8f0d9e531f88c3e"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.662257 31335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance:
uuid: "68eef2375f614bc786a25be214145a0f"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "46681ecc52e50a39575bd575b783e361"
server_key_iv: "399f223aeb5a6e6ab8f0d9e531f88c3e"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.663614 31335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:18.664424 31349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.664609 31335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:02:18.664740 31335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal
uuid: "68eef2375f614bc786a25be214145a0f"
format_stamp: "Formatted at 2026-09-28 13:02:18 on dist-test-slave-8sjz"
server_key: "46681ecc52e50a39575bd575b783e361"
server_key_iv: "399f223aeb5a6e6ab8f0d9e531f88c3e"
server_key_version: "encryptionkey@0"
I20260928 13:02:18.664825 31335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:18.677779 31335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:18.678117 31335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:18.678288 31335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:18.678531 31335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:02:18.678869 31335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:02:18.678925 31335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.678964 31335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:02:18.678992 31335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:18.685000 31335 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.67:35509
I20260928 13:02:18.685075 31462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.67:35509 every 8 connection(s)
I20260928 13:02:18.685429 31335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/data/info.pb
I20260928 13:02:18.690809 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.686216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:38871 (local address 127.11.11.126:46775)
0928 13:02:18.686287 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.686289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.686526 (+ 237us) server_negotiation.cc:407] Connection header received
0928 13:02:18.687000 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.687001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.687011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.687064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.687721 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.688390 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.689392 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.689593 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.689665 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.690560 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.690561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.690562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:18.690566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.690582 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:18.690586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.690642 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.690714 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.690742 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:02:18.691015 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.686203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46775 (local address 127.11.11.67:38871)
0928 13:02:18.686386 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.686421 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.686914 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.687121 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.687125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.687134 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:18.687633 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.687637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.688467 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.688468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.689288 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.689290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.689678 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.689679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.689791 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.690417 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:02:18.690418 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:18.690447 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:02:18.690471 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:18.690642 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:18.690644 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:18.690647 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.690698 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.690778 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260928 13:02:18.691134 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31335
I20260928 13:02:18.691248 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal/instance
I20260928 13:02:18.691478 11309 external_mini_cluster.cc:1428] Setting key 6c4234e678cf20137d71ff5f9da9c94b
I20260928 13:02:18.691516 31463 heartbeater.cc:347] Connected to a master server at 127.11.11.126:46775
I20260928 13:02:18.691610 31463 heartbeater.cc:464] Registering TS with master...
I20260928 13:02:18.691839 31463 heartbeater.cc:511] Master 127.11.11.126:46775 requested a full tablet report, sending...
I20260928 13:02:18.692319 31012 ts_manager.cc:194] Registered new tserver with Master: 68eef2375f614bc786a25be214145a0f (127.11.11.67:35509)
I20260928 13:02:18.692742 31012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.11.67:38871
I20260928 13:02:18.703488 11309 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:02:18.707852 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.705107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.11.11.126:46775)
0928 13:02:18.705175 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.705177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.705218 (+ 41us) server_negotiation.cc:407] Connection header received
0928 13:02:18.705281 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.705282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.705293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.705349 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:18.705488 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.705980 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.706715 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.706897 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.706953 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.707641 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:18.707642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:18.707642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:18.707647 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:18.707660 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:18.707664 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:18.707713 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.707764 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.707789 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:02:18.709668 31012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:02:18.711290 31012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:02:18.718087 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.713662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.11.11.65:42179)
0928 13:02:18.713843 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.713849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.713886 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:02:18.713997 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.714000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.714015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.714089 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.714332 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715534 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717017 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.717693 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717758 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.717852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.717965 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.718003 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260928 13:02:18.718113 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.713549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:42179 (local address 127.0.0.1:38394)
0928 13:02:18.713787 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.713833 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.713911 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.714100 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.714117 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.714119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.714188 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.714190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715620 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.715621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.716942 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.716945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.717773 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.717858 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.717875 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.717952 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.718015 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":102,"thread_start_us":57,"threads_started":1}
I20260928 13:02:18.718400 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.714155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.11.11.67:35509)
0928 13:02:18.714341 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.714345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.714466 (+ 121us) server_negotiation.cc:407] Connection header received
0928 13:02:18.714538 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.714541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.714557 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.714635 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.714739 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715763 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.716914 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.717672 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.717852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.718276 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.718317 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260928 13:02:18.718967 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.713557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36249 (local address 127.0.0.1:43610)
0928 13:02:18.713873 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.713885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.713935 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.714560 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.714560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.714562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.714605 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.714606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715512 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.715513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.716942 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.716945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.718748 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.718749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.718809 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.718812 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.718857 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.718895 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":35,"thread_start_us":168,"threads_started":1}
I20260928 13:02:18.718977 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.713899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.11.11.66:36249)
0928 13:02:18.714216 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.714223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.714233 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:02:18.714273 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.714275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.714292 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.714685 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.714748 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715422 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717890 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.718672 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.718747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.718838 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.718866 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.718901 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":79,"threads_started":1}
I20260928 13:02:18.718904 31135 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 (DEFAULT_TABLE table=test-table [id=13cf218555d044d28f4f33000244385a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.719291 31135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ad12892cb47d8982a503ed781a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.718113 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.713754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.67:35509 (local address 127.0.0.1:45122)
0928 13:02:18.714404 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.714414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.714464 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.714646 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.714646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.714648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.714686 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.714687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.715848 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.715849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.716827 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.716829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.717772 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.717773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.717858 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.717875 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.717952 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.718015 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260928 13:02:18.720736 31266 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 (DEFAULT_TABLE table=test-table [id=13cf218555d044d28f4f33000244385a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.721052 31266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ad12892cb47d8982a503ed781a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.722429 31477 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: Bootstrap starting.
I20260928 13:02:18.723203 31477 tablet_bootstrap.cc:654] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.723532 31477 log.cc:824] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.724380 31477 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: No bootstrap required, opened a new log
I20260928 13:02:18.724455 31477 ts_tablet_manager.cc:1397] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:18.724634 31397 tablet_service.cc:1467] Processing CreateTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 (DEFAULT_TABLE table=test-table [id=13cf218555d044d28f4f33000244385a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:02:18.724973 31397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7ad12892cb47d8982a503ed781a2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:18.726451 31477 raft_consensus.cc:348] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.726573 31477 raft_consensus.cc:374] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.726594 31477 raft_consensus.cc:729] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86809474ffa14feb81ac3721491b98e7, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.726722 31477 consensus_queue.cc:260] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.726994 31477 ts_tablet_manager.cc:1428] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:02:18.727051 31201 heartbeater.cc:503] Master 127.11.11.126:46775 was elected leader, sending a full tablet report...
I20260928 13:02:18.727941 31481 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Bootstrap starting.
I20260928 13:02:18.728348 31479 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Bootstrap starting.
I20260928 13:02:18.728586 31481 tablet_bootstrap.cc:654] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.728884 31481 log.cc:824] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.728936 31479 tablet_bootstrap.cc:654] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:18.729254 31479 log.cc:824] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:18.729737 31481 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: No bootstrap required, opened a new log
I20260928 13:02:18.729804 31481 ts_tablet_manager.cc:1397] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:18.729992 31479 tablet_bootstrap.cc:492] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: No bootstrap required, opened a new log
I20260928 13:02:18.730070 31479 ts_tablet_manager.cc:1397] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:02:18.731518 31481 raft_consensus.cc:348] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.731657 31481 raft_consensus.cc:374] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.731679 31481 raft_consensus.cc:729] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68eef2375f614bc786a25be214145a0f, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.731802 31481 consensus_queue.cc:260] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.732105 31481 ts_tablet_manager.cc:1428] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:02:18.732142 31463 heartbeater.cc:503] Master 127.11.11.126:46775 was elected leader, sending a full tablet report...
I20260928 13:02:18.732028 31479 raft_consensus.cc:348] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.732219 31479 raft_consensus.cc:374] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:18.732255 31479 raft_consensus.cc:729] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956de805d8b49ba9e1e94e9ed16e0fd, State: Initialized, Role: FOLLOWER
I20260928 13:02:18.732364 31479 consensus_queue.cc:260] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.732625 31479 ts_tablet_manager.cc:1428] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:02:18.732782 31332 heartbeater.cc:503] Master 127.11.11.126:46775 was elected leader, sending a full tablet report...
W20260928 13:02:18.812219 31333 tablet.cc:2369] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:02:18.870896 31484 raft_consensus.cc:482] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:02:18.871018 31484 raft_consensus.cc:504] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.871373 31484 leader_election.cc:288] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a956de805d8b49ba9e1e94e9ed16e0fd (127.11.11.66:36249), 86809474ffa14feb81ac3721491b98e7 (127.11.11.65:42179)
I20260928 13:02:18.874349 31475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.871632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:35533 (local address 127.11.11.66:36249)
0928 13:02:18.871681 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.871684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.871695 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:18.871792 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.871794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.871805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.871871 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.872069 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.872582 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.873581 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.873976 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.874030 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.874036 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.874222 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.874271 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:02:18.874388 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.871681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.11.67:52017 (local address 127.11.11.65:42179)
0928 13:02:18.871736 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.871739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.871877 (+ 138us) server_negotiation.cc:407] Connection header received
0928 13:02:18.871920 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.871920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.871933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.871987 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:02:18.872185 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.872711 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.873645 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.873995 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.874099 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.874106 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.874277 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.874322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:02:18.874385 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.871556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.66:36249 (local address 127.11.11.67:35533)
0928 13:02:18.871632 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.871646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.871695 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.871888 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.871889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.871891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.871969 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.871972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.872662 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.872663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.873489 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.873492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.874055 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.874056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.874127 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.874147 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.874198 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.874252 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 13:02:18.874594 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.871581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:42179 (local address 127.11.11.67:52017)
0928 13:02:18.871790 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.871800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:18.871863 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:18.871998 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:18.871999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:18.872000 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:02:18.872107 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.872109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.872806 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.872807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.873520 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:18.873522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.874106 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.874107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:18.874165 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.874175 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 13:02:18.874252 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 13:02:18.874287 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":134,"threads_started":1}
I20260928 13:02:18.874723 31286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ad12892cb47d8982a503ed781a2a6" candidate_uuid: "68eef2375f614bc786a25be214145a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" is_pre_election: true
I20260928 13:02:18.874910 31286 raft_consensus.cc:2454] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68eef2375f614bc786a25be214145a0f in term 0.
I20260928 13:02:18.874974 31155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ad12892cb47d8982a503ed781a2a6" candidate_uuid: "68eef2375f614bc786a25be214145a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86809474ffa14feb81ac3721491b98e7" is_pre_election: true
I20260928 13:02:18.875152 31155 raft_consensus.cc:2454] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68eef2375f614bc786a25be214145a0f in term 0.
I20260928 13:02:18.875161 31353 leader_election.cc:302] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68eef2375f614bc786a25be214145a0f, a956de805d8b49ba9e1e94e9ed16e0fd; no voters:
I20260928 13:02:18.875319 31484 raft_consensus.cc:2781] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:02:18.875367 31484 raft_consensus.cc:482] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:02:18.875402 31484 raft_consensus.cc:3037] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.876165 31484 raft_consensus.cc:504] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.876307 31484 leader_election.cc:288] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [CANDIDATE]: Term 1 election: Requested vote from peers a956de805d8b49ba9e1e94e9ed16e0fd (127.11.11.66:36249), 86809474ffa14feb81ac3721491b98e7 (127.11.11.65:42179)
I20260928 13:02:18.876459 31286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ad12892cb47d8982a503ed781a2a6" candidate_uuid: "68eef2375f614bc786a25be214145a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd"
I20260928 13:02:18.876459 31155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7ad12892cb47d8982a503ed781a2a6" candidate_uuid: "68eef2375f614bc786a25be214145a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86809474ffa14feb81ac3721491b98e7"
I20260928 13:02:18.876540 31155 raft_consensus.cc:3037] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.876540 31286 raft_consensus.cc:3037] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:18.877460 31286 raft_consensus.cc:2454] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eef2375f614bc786a25be214145a0f in term 1.
I20260928 13:02:18.877454 31155 raft_consensus.cc:2454] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68eef2375f614bc786a25be214145a0f in term 1.
I20260928 13:02:18.877629 31353 leader_election.cc:302] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68eef2375f614bc786a25be214145a0f, a956de805d8b49ba9e1e94e9ed16e0fd; no voters:
I20260928 13:02:18.877741 31484 raft_consensus.cc:2781] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:18.877883 31484 raft_consensus.cc:686] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 1 LEADER]: Becoming Leader. State: Replica: 68eef2375f614bc786a25be214145a0f, State: Running, Role: LEADER
I20260928 13:02:18.877990 31484 consensus_queue.cc:237] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } }
I20260928 13:02:18.878895 31013 catalog_manager.cc:5495] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 68eef2375f614bc786a25be214145a0f (127.11.11.67). New cstate: current_term: 1 leader_uuid: "68eef2375f614bc786a25be214145a0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68eef2375f614bc786a25be214145a0f" member_type: VOTER last_known_addr { host: "127.11.11.67" port: 35509 } health_report { overall_health: HEALTHY } } }
I20260928 13:02:18.898007 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.895739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.11.11.67:35509)
0928 13:02:18.895782 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:18.895785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:18.895797 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:02:18.895873 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:18.895874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:18.895891 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:18.895958 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:18.896078 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.896550 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.897318 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:18.897503 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:18.897552 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:18.897710 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:18.897812 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:18.897875 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:18.897912 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 13:02:18.897928 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:02:18.899856 31155 raft_consensus.cc:1261] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 1 FOLLOWER]: Refusing update from remote peer 68eef2375f614bc786a25be214145a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:18.899856 31286 raft_consensus.cc:1261] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 1 FOLLOWER]: Refusing update from remote peer 68eef2375f614bc786a25be214145a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:18.900242 31484 consensus_queue.cc:1035] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956de805d8b49ba9e1e94e9ed16e0fd" member_type: VOTER last_known_addr { host: "127.11.11.66" port: 36249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:18.900382 31487 consensus_queue.cc:1035] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86809474ffa14feb81ac3721491b98e7" member_type: VOTER last_known_addr { host: "127.11.11.65" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:18.903028 31492 mvcc.cc:205] Tried to move back new op lower bound from 7334299807330942976 to 7334299807244988416. Current Snapshot: MvccSnapshot[applied={T|T < 7334299807330942976}]
I20260928 13:02:18.902697 31490 mvcc.cc:205] Tried to move back new op lower bound from 7334299807330942976 to 7334299807244988416. Current Snapshot: MvccSnapshot[applied={T|T < 7334299807330942976}]
I20260928 13:02:18.903487 31491 mvcc.cc:205] Tried to move back new op lower bound from 7334299807330942976 to 7334299807244988416. Current Snapshot: MvccSnapshot[applied={T|T < 7334299807330942976}]
I20260928 13:02:18.905644 31012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:18.905751 31012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:02:18.906538 31012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22c577c5b3ea43f5a5af42b347483b9d: Sending DeleteTablet for 3 replicas of tablet eb7ad12892cb47d8982a503ed781a2a6
I20260928 13:02:18.906913 31266 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='slave'} at 127.0.0.1:43610
I20260928 13:02:18.906914 31397 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='slave'} at 127.0.0.1:45122
I20260928 13:02:18.907141 31135 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7ad12892cb47d8982a503ed781a2a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:02:18 UTC) from {username='slave'} at 127.0.0.1:38394
I20260928 13:02:18.907176 31499 tablet_replica.cc:330] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: stopping tablet replica
I20260928 13:02:18.907280 31499 raft_consensus.cc:2229] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:18.907397 31499 raft_consensus.cc:2258] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:18.907409 31501 tablet_replica.cc:330] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7: stopping tablet replica
I20260928 13:02:18.907557 31500 tablet_replica.cc:330] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: stopping tablet replica
I20260928 13:02:18.907620 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31073
I20260928 13:02:18.907583 31501 raft_consensus.cc:2229] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:02:18.907658 31500 raft_consensus.cc:2229] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:02:18.907763 31500 raft_consensus.cc:2258] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:18.907855 31501 raft_consensus.cc:2258] T eb7ad12892cb47d8982a503ed781a2a6 P 86809474ffa14feb81ac3721491b98e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:02:18.908917 31499 ts_tablet_manager.cc:1905] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:18.910427 31499 ts_tablet_manager.cc:1918] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:18.910522 31499 log.cc:1196] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-1/wal/wals/eb7ad12892cb47d8982a503ed781a2a6
I20260928 13:02:18.910789 31499 ts_tablet_manager.cc:1939] T eb7ad12892cb47d8982a503ed781a2a6 P a956de805d8b49ba9e1e94e9ed16e0fd: Deleting consensus metadata
I20260928 13:02:18.911243 31000 catalog_manager.cc:4841] TS a956de805d8b49ba9e1e94e9ed16e0fd (127.11.11.66:36249): tablet eb7ad12892cb47d8982a503ed781a2a6 (table test-table [id=13cf218555d044d28f4f33000244385a]) successfully deleted
W20260928 13:02:18.913587 30998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.11.65:42179 (error 108)
I20260928 13:02:18.913781 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31204
I20260928 13:02:18.914095 31474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:18.913969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.65:42179 (local address 127.0.0.1:38410)
0928 13:02:18.914020 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:18.914057 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.65:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:02:18.914201 30998 catalog_manager.cc:4584] TS 86809474ffa14feb81ac3721491b98e7 (127.11.11.65:42179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb7ad12892cb47d8982a503ed781a2a6: Network error: Client connection negotiation failed: client connection to 127.11.11.65:42179: connect: Connection refused (error 111)
I20260928 13:02:18.914570 31500 ts_tablet_manager.cc:1905] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:02:18.916688 31500 ts_tablet_manager.cc:1918] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:02:18.916774 31500 log.cc:1196] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Deleting WAL directory at /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600290352246-11309-0/minicluster-data/ts-2/wal/wals/eb7ad12892cb47d8982a503ed781a2a6
I20260928 13:02:18.917034 31500 ts_tablet_manager.cc:1939] T eb7ad12892cb47d8982a503ed781a2a6 P 68eef2375f614bc786a25be214145a0f: Deleting consensus metadata
I20260928 13:02:18.917626 30998 catalog_manager.cc:4841] TS 68eef2375f614bc786a25be214145a0f (127.11.11.67:35509): tablet eb7ad12892cb47d8982a503ed781a2a6 (table test-table [id=13cf218555d044d28f4f33000244385a]) successfully deleted
I20260928 13:02:18.920830 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31335
I20260928 13:02:18.925899 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 30982
2026-09-28T13:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (772 ms)
[----------] 6 tests from EncryptionPolicyTest (8608 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:18 dist-test-slave-8sjz krb5kdc[31505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:18 dist-test-slave-8sjz krb5kdc[31505](info): set up 2 sockets
Sep 28 13:02:18 dist-test-slave-8sjz krb5kdc[31505](info): commencing operation
krb5kdc: starting...
W20260928 13:02:19.969115 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:20.091199 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:20.173363 31518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:20.173569 31518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:20.173591 31518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:20.174768 31518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:20.174817 31518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:20.174831 31518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:20.174844 31518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:20.174856 31518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:20.176489 31518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:20.176748 31518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:20.176990 31518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:20.181946 31523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:20.182029 31518 server_base.cc:1034] running on GCE node
W20260928 13:02:20.181941 31524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:20.181941 31526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:20.182480 31518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:20.182783 31518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:20.183946 31518 hybrid_clock.cc:648] HybridClock initialized: now 1790600540183942 us; error 34 us; skew 500 ppm
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:20.186805 31518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:20.187247 31518 webserver.cc:460] Webserver started at http://127.0.0.1:38671/ using document root <none> and password file <none>
I20260928 13:02:20.187453 31518 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:20.187495 31518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:20.187584 31518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:20.188752 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "caef04b3fe994ae0b8a4c9ea623ade7e"
format_stamp: "Formatted at 2026-09-28 13:02:20 on dist-test-slave-8sjz"
server_key: "17b9b41ae767ce545f7468a2a8fa9d87"
server_key_iv: "e96d7556f11558e3057bc6d6748c103a"
server_key_version: "encryptionkey@0"
I20260928 13:02:20.189055 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "caef04b3fe994ae0b8a4c9ea623ade7e"
format_stamp: "Formatted at 2026-09-28 13:02:20 on dist-test-slave-8sjz"
server_key: "17b9b41ae767ce545f7468a2a8fa9d87"
server_key_iv: "e96d7556f11558e3057bc6d6748c103a"
server_key_version: "encryptionkey@0"
I20260928 13:02:20.190308 31518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:20.191020 31533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:20.191200 31518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:20.191282 31518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "caef04b3fe994ae0b8a4c9ea623ade7e"
format_stamp: "Formatted at 2026-09-28 13:02:20 on dist-test-slave-8sjz"
server_key: "17b9b41ae767ce545f7468a2a8fa9d87"
server_key_iv: "e96d7556f11558e3057bc6d6748c103a"
server_key_version: "encryptionkey@0"
I20260928 13:02:20.191335 31518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:20.198068 31518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:20.200335 31518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:20.200448 31518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:20.204104 31518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32977
I20260928 13:02:20.204129 31585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32977 every 8 connection(s)
I20260928 13:02:20.204500 31518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:20.205334 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:20.206003 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31518
I20260928 13:02:20.206118 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:20.206318 11309 external_mini_cluster.cc:1428] Setting key 3d939e30cd4de47e755e428882d0b7ad
I20260928 13:02:20.208190 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e: Bootstrap starting.
I20260928 13:02:20.208766 31586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:20.209051 31586 log.cc:824] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:20.209734 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e: No bootstrap required, opened a new log
I20260928 13:02:20.211498 31586 raft_consensus.cc:348] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260928 13:02:20.211654 31586 raft_consensus.cc:374] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:20.211701 31586 raft_consensus.cc:729] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caef04b3fe994ae0b8a4c9ea623ade7e, State: Initialized, Role: FOLLOWER
I20260928 13:02:20.211869 31586 consensus_queue.cc:260] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260928 13:02:20.211962 31586 raft_consensus.cc:388] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:20.212006 31586 raft_consensus.cc:482] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:20.212072 31586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:20.212711 31586 raft_consensus.cc:504] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260928 13:02:20.212834 31586 leader_election.cc:302] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caef04b3fe994ae0b8a4c9ea623ade7e; no voters:
I20260928 13:02:20.212992 31586 leader_election.cc:288] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:02:20.213068 31591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:20.213210 31591 raft_consensus.cc:686] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [term 1 LEADER]: Becoming Leader. State: Replica: caef04b3fe994ae0b8a4c9ea623ade7e, State: Running, Role: LEADER
I20260928 13:02:20.213255 31586 sys_catalog.cc:564] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:20.213285 31591 consensus_queue.cc:237] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260928 13:02:20.213554 31591 sys_catalog.cc:455] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader caef04b3fe994ae0b8a4c9ea623ade7e. Latest consensus state: current_term: 1 leader_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260928 13:02:20.213631 31591 sys_catalog.cc:458] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:20.213888 31597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:20.214430 31597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:20.213565 31592 sys_catalog.cc:455] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caef04b3fe994ae0b8a4c9ea623ade7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260928 13:02:20.215615 31592 sys_catalog.cc:458] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:20.216459 31597 catalog_manager.cc:1347] Generated new cluster ID: 01811ca5faba45869dc8c866534e9f50
I20260928 13:02:20.216511 31597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:20.217651 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:20.206860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56227 (local address 127.0.0.1:32977)
0928 13:02:20.207051 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:20.207060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:20.207085 (+ 25us) server_negotiation.cc:407] Connection header received
0928 13:02:20.207506 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:20.207512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:20.207558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:20.207703 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:20.208478 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:20.209465 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:20.210282 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:20.210713 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:20.210825 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:20.213636 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:20.213639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:20.213640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:20.213652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:20.215784 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:20.216321 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:20.216324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:20.216324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:20.216436 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:20.216736 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:20.216737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:20.216737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:20.217023 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:20.217166 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:20.217343 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 13:02:20.217439 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260928 13:02:20.221946 31597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:20.222553 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:20.222643 31597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:20.230195 31597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caef04b3fe994ae0b8a4c9ea623ade7e: Generated new TSK 0
I20260928 13:02:20.230399 31597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:20.236150 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:20.224334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43243 (local address 127.0.0.1:32977)
0928 13:02:20.224377 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:20.224379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:20.229364 (+ 4985us) server_negotiation.cc:407] Connection header received
0928 13:02:20.229446 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:20.229447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:20.229458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:20.229513 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:20.230157 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:20.231003 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:20.231755 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:20.232005 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:20.232148 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:20.233424 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:20.233426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:20.233427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:20.233434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:20.234885 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:20.235268 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:20.235270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:20.235270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:20.235332 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:20.235566 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:20.235567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:20.235567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:20.235694 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:20.235814 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:20.235935 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 13:02:20.236002 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:20.238476 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31518
W20260928 13:02:20.242691 31503 connection.cc:535] client connection to 127.0.0.1:32977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32977): Connection reset by peer (error 104)
2026-09-28T13:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31617](info): set up 2 sockets
Sep 28 13:02:20 dist-test-slave-8sjz krb5kdc[31617](info): commencing operation
krb5kdc: starting...
W20260928 13:02:21.293391 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:21 dist-test-slave-8sjz krb5kdc[31617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:21.419847 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:21.502717 31630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:21.502990 31630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:21.503031 31630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:21.504356 31630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:21.504422 31630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:21.504447 31630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:21.504477 31630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:21.504501 31630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:21.506088 31630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:21.506395 31630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:21.506630 31630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:21.511665 31635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:21.511677 31638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:21.511755 31636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:21.511896 31630 server_base.cc:1034] running on GCE node
I20260928 13:02:21.512218 31630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:21.512521 31630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:21.513680 31630 hybrid_clock.cc:648] HybridClock initialized: now 1790600541513659 us; error 52 us; skew 500 ppm
Sep 28 13:02:21 dist-test-slave-8sjz krb5kdc[31617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:21.522289 31630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:21.522825 31630 webserver.cc:460] Webserver started at http://127.0.0.1:41511/ using document root <none> and password file <none>
I20260928 13:02:21.523046 31630 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:21.523090 31630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:21.523195 31630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:21.524362 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "ff2720ff0d094a25bd4beb2ed6b1b896"
format_stamp: "Formatted at 2026-09-28 13:02:21 on dist-test-slave-8sjz"
server_key: "f5bcb0e6d67b8293f955656d527b7563"
server_key_iv: "c23c75816ae0a59643f667a6e2848304"
server_key_version: "encryptionkey@0"
I20260928 13:02:21.524751 31630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "ff2720ff0d094a25bd4beb2ed6b1b896"
format_stamp: "Formatted at 2026-09-28 13:02:21 on dist-test-slave-8sjz"
server_key: "f5bcb0e6d67b8293f955656d527b7563"
server_key_iv: "c23c75816ae0a59643f667a6e2848304"
server_key_version: "encryptionkey@0"
I20260928 13:02:21.526105 31630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:02:21.526851 31645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:21.527051 31630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:21.527156 31630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "ff2720ff0d094a25bd4beb2ed6b1b896"
format_stamp: "Formatted at 2026-09-28 13:02:21 on dist-test-slave-8sjz"
server_key: "f5bcb0e6d67b8293f955656d527b7563"
server_key_iv: "c23c75816ae0a59643f667a6e2848304"
server_key_version: "encryptionkey@0"
I20260928 13:02:21.527249 31630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:21.542135 31630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:21.544281 31630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:21.544454 31630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:21.548777 31630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36115
I20260928 13:02:21.548836 31697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36115 every 8 connection(s)
I20260928 13:02:21.549224 31630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:21.550086 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:21.552420 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896: Bootstrap starting.
I20260928 13:02:21.553006 31698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:21.553287 31698 log.cc:824] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:21.554054 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896: No bootstrap required, opened a new log
I20260928 13:02:21.555235 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31630
I20260928 13:02:21.555348 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:21.555575 11309 external_mini_cluster.cc:1428] Setting key df969accfc51a8b9d37f4f4778515f49
I20260928 13:02:21.556193 31698 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 13:02:21.556380 31698 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:21.556406 31698 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2720ff0d094a25bd4beb2ed6b1b896, State: Initialized, Role: FOLLOWER
I20260928 13:02:21.556555 31698 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 13:02:21.556624 31698 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:21.556648 31698 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:21.556713 31698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:21.557420 31698 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 13:02:21.557538 31698 leader_election.cc:302] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff2720ff0d094a25bd4beb2ed6b1b896; no voters:
I20260928 13:02:21.557720 31698 leader_election.cc:288] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:21.557749 31703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:21.557864 31703 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [term 1 LEADER]: Becoming Leader. State: Replica: ff2720ff0d094a25bd4beb2ed6b1b896, State: Running, Role: LEADER
I20260928 13:02:21.557991 31698 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:21.557999 31703 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } }
I20260928 13:02:21.558396 31705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff2720ff0d094a25bd4beb2ed6b1b896. Latest consensus state: current_term: 1 leader_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260928 13:02:21.558511 31705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:21.558724 31704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2720ff0d094a25bd4beb2ed6b1b896" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36115 } } }
I20260928 13:02:21.558835 31704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:21.558990 31711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:21.559499 31711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:21.561122 31711 catalog_manager.cc:1347] Generated new cluster ID: 1a81fbfa21444bb78e35fd7984ec7138
I20260928 13:02:21.561182 31711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:21.562109 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:21.556127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53783 (local address 127.0.0.1:36115)
0928 13:02:21.556303 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:21.556312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:21.556341 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:02:21.556488 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:21.556496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:21.556556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:21.556716 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:21.556983 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:21.558517 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:21.559626 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:21.560362 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:21.560493 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:21.561429 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:21.561433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:21.561433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:21.561636 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:21.561675 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:21.561690 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:21.561780 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:21.561872 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 13:02:21.561902 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 13:02:21.563067 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:21.568137 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:21.564608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50821 (local address 127.0.0.1:36115)
0928 13:02:21.564641 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:21.564643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:21.564739 (+ 96us) server_negotiation.cc:407] Connection header received
0928 13:02:21.564768 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:21.564769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:21.564778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:21.564847 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:21.564952 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:21.565695 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:21.566687 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:21.567053 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:21.567139 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:21.567856 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:21.567858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:21.567858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:21.567863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:21.567880 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:21.567885 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:21.567944 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:21.568024 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 13:02:21.568059 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 13:02:21.575256 31711 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:21.575999 31711 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:21.581768 31711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff2720ff0d094a25bd4beb2ed6b1b896: Generated new TSK 0
I20260928 13:02:21.582017 31711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:21.587273 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31630
W20260928 13:02:21.591629 31615 connection.cc:535] client connection to 127.0.0.1:36115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36115): Connection reset by peer (error 104)
2026-09-28T13:02:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:21 dist-test-slave-8sjz krb5kdc[31729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:21 dist-test-slave-8sjz krb5kdc[31729](info): set up 2 sockets
Sep 28 13:02:21 dist-test-slave-8sjz krb5kdc[31729](info): commencing operation
krb5kdc: starting...
W20260928 13:02:22.629132 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:22 dist-test-slave-8sjz krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:22.757548 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:22.849469 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:22.849742 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:22.849772 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:22.851101 31742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:22.851152 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:22.851167 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:22.851181 31742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:22.851193 31742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:22.852867 31742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:22.853088 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:22.853271 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:22.858356 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:22.858273 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:22.858480 31742 server_base.cc:1034] running on GCE node
W20260928 13:02:22.858275 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:22.858866 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:22.859130 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:22.860275 31742 hybrid_clock.cc:648] HybridClock initialized: now 1790600542860239 us; error 54 us; skew 500 ppm
Sep 28 13:02:22 dist-test-slave-8sjz krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:22.863451 31742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:22.864133 31742 webserver.cc:460] Webserver started at http://127.0.0.1:36009/ using document root <none> and password file <none>
I20260928 13:02:22.864457 31742 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:22.864533 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:22.864705 31742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:22.866171 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "a5ab8b6cc9eb4c67abefe511435ba16b"
format_stamp: "Formatted at 2026-09-28 13:02:22 on dist-test-slave-8sjz"
server_key: "963cb076245cd800f246188e18a75e9d"
server_key_iv: "89ef3c86b247eacd778a300d9d57ac5e"
server_key_version: "encryptionkey@0"
I20260928 13:02:22.866492 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "a5ab8b6cc9eb4c67abefe511435ba16b"
format_stamp: "Formatted at 2026-09-28 13:02:22 on dist-test-slave-8sjz"
server_key: "963cb076245cd800f246188e18a75e9d"
server_key_iv: "89ef3c86b247eacd778a300d9d57ac5e"
server_key_version: "encryptionkey@0"
I20260928 13:02:22.867830 31742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:22.868671 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:22.868860 31742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:02:22.868955 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "a5ab8b6cc9eb4c67abefe511435ba16b"
format_stamp: "Formatted at 2026-09-28 13:02:22 on dist-test-slave-8sjz"
server_key: "963cb076245cd800f246188e18a75e9d"
server_key_iv: "89ef3c86b247eacd778a300d9d57ac5e"
server_key_version: "encryptionkey@0"
I20260928 13:02:22.869050 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:22.872665 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:22.875101 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:22.875278 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:22.879324 31742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45633
I20260928 13:02:22.879359 31809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45633 every 8 connection(s)
I20260928 13:02:22.879722 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:22.880843 31810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:22.883054 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31742
I20260928 13:02:22.883140 31810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b: Bootstrap starting.
I20260928 13:02:22.883199 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:22.883397 11309 external_mini_cluster.cc:1428] Setting key bc169a5c0e76f22ad86c32a4328d74b7
I20260928 13:02:22.883891 31810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:22.884287 31810 log.cc:824] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:22.885044 31810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b: No bootstrap required, opened a new log
I20260928 13:02:22.886106 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:22.883958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47263 (local address 127.0.0.1:45633)
0928 13:02:22.884186 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:22.884195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:22.884222 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:02:22.884766 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:22.884774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:22.884804 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:22.884932 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:22.885596 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:22.885599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:22.885599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:22.885603 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:22.885641 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:22.885651 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:22.885773 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:22.885843 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 13:02:22.885901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260928 13:02:22.887156 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:22.887523 31810 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } }
I20260928 13:02:22.887662 31810 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:22.887686 31810 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ab8b6cc9eb4c67abefe511435ba16b, State: Initialized, Role: FOLLOWER
I20260928 13:02:22.887835 31810 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } }
I20260928 13:02:22.887914 31810 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:22.887938 31810 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:22.888006 31810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:22.888703 31810 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } }
I20260928 13:02:22.888821 31810 leader_election.cc:302] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ab8b6cc9eb4c67abefe511435ba16b; no voters:
I20260928 13:02:22.888979 31810 leader_election.cc:288] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:22.889004 31820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:22.889182 31820 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [term 1 LEADER]: Becoming Leader. State: Replica: a5ab8b6cc9eb4c67abefe511435ba16b, State: Running, Role: LEADER
I20260928 13:02:22.889290 31810 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:22.889317 31820 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } }
I20260928 13:02:22.889667 31821 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } } }
I20260928 13:02:22.889649 31820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ab8b6cc9eb4c67abefe511435ba16b. Latest consensus state: current_term: 1 leader_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ab8b6cc9eb4c67abefe511435ba16b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45633 } } }
I20260928 13:02:22.889878 31820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:22.890022 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:22.888792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53471 (local address 127.0.0.1:45633)
0928 13:02:22.888833 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:22.888835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:22.888962 (+ 127us) server_negotiation.cc:407] Connection header received
0928 13:02:22.889037 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:22.889038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:22.889043 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:22.889092 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:22.889747 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:22.889748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:22.889749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:22.889753 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:22.889768 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:22.889772 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:22.889804 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:22.889928 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 13:02:22.889942 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:02:22.890127 31829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:22.889864 31821 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:22.890677 31829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:22.893249 31829 catalog_manager.cc:1347] Generated new cluster ID: 55f0cdd0db444fde81b3bed5fe8b59c0
I20260928 13:02:22.893306 31829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:22.909425 31829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:22.909641 31829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:22.917810 31829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ab8b6cc9eb4c67abefe511435ba16b: Generated new TSK 0
I20260928 13:02:22.918079 31829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:22.926959 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31742
2026-09-28T13:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:22 dist-test-slave-8sjz krb5kdc[31841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:22 dist-test-slave-8sjz krb5kdc[31841](info): set up 2 sockets
Sep 28 13:02:22 dist-test-slave-8sjz krb5kdc[31841](info): commencing operation
krb5kdc: starting...
W20260928 13:02:23.965137 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:24.092628 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:24.190896 31854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:24.191116 31854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:24.191141 31854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:24.192268 31854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:24.192304 31854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:24.192318 31854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:24.192332 31854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:24.192342 31854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:24.193749 31854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:24.193943 31854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:24.194113 31854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:24.199076 31859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:24.199091 31860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:24.199091 31862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:24.199482 31854 server_base.cc:1034] running on GCE node
I20260928 13:02:24.199735 31854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:24.200083 31854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:24.201278 31854 hybrid_clock.cc:648] HybridClock initialized: now 1790600544201261 us; error 41 us; skew 500 ppm
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:24.204083 31854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:24.204515 31854 webserver.cc:460] Webserver started at http://127.0.0.1:42825/ using document root <none> and password file <none>
I20260928 13:02:24.204717 31854 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:24.204761 31854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:24.204850 31854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:24.205893 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "29b54afe346b426cbab569eacd0d1860"
format_stamp: "Formatted at 2026-09-28 13:02:24 on dist-test-slave-8sjz"
server_key: "c6de2859532b88ff679e6a53573ba37a"
server_key_iv: "2b7648d641ad273668b31f2044ab3619"
server_key_version: "encryptionkey@0"
I20260928 13:02:24.206198 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "29b54afe346b426cbab569eacd0d1860"
format_stamp: "Formatted at 2026-09-28 13:02:24 on dist-test-slave-8sjz"
server_key: "c6de2859532b88ff679e6a53573ba37a"
server_key_iv: "2b7648d641ad273668b31f2044ab3619"
server_key_version: "encryptionkey@0"
I20260928 13:02:24.207513 31854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:02:24.208364 31869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:24.208536 31854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:24.208630 31854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "29b54afe346b426cbab569eacd0d1860"
format_stamp: "Formatted at 2026-09-28 13:02:24 on dist-test-slave-8sjz"
server_key: "c6de2859532b88ff679e6a53573ba37a"
server_key_iv: "2b7648d641ad273668b31f2044ab3619"
server_key_version: "encryptionkey@0"
I20260928 13:02:24.208689 31854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:24.227059 31854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:24.229244 31854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:24.229368 31854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:24.233291 31854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42565
I20260928 13:02:24.233346 31921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42565 every 8 connection(s)
I20260928 13:02:24.233721 31854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:24.234642 31922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:24.237118 31922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: Bootstrap starting.
I20260928 13:02:24.237730 31922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:24.238027 31922 log.cc:824] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:24.238816 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31854
I20260928 13:02:24.238837 31922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: No bootstrap required, opened a new log
I20260928 13:02:24.238992 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:24.239233 11309 external_mini_cluster.cc:1428] Setting key ecf402737901a2d54db440797d118950
I20260928 13:02:24.241235 31922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260928 13:02:24.241459 31922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:24.241496 31922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b54afe346b426cbab569eacd0d1860, State: Initialized, Role: FOLLOWER
I20260928 13:02:24.241624 31922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260928 13:02:24.241720 31922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:24.241813 31922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:24.241863 31922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:24.242525 31922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260928 13:02:24.242635 31922 leader_election.cc:302] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b54afe346b426cbab569eacd0d1860; no voters:
I20260928 13:02:24.242807 31922 leader_election.cc:288] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:24.242897 31927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:24.243060 31922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:24.243091 31927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [term 1 LEADER]: Becoming Leader. State: Replica: 29b54afe346b426cbab569eacd0d1860, State: Running, Role: LEADER
I20260928 13:02:24.243306 31927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } }
I20260928 13:02:24.243686 31928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b54afe346b426cbab569eacd0d1860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } } }
I20260928 13:02:24.243767 31928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:24.243939 31929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b54afe346b426cbab569eacd0d1860. Latest consensus state: current_term: 1 leader_uuid: "29b54afe346b426cbab569eacd0d1860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b54afe346b426cbab569eacd0d1860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42565 } } }
I20260928 13:02:24.243999 31929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860 [sys.catalog]: This master's current role is: LEADER
W20260928 13:02:24.244982 31943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:24.245042 31943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:24.245085 31936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:24.245285 31936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:24.246866 31936 catalog_manager.cc:1347] Generated new cluster ID: 2f0b699c65f54644a1f9f30100290e48
I20260928 13:02:24.246922 31936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:02:24.251962 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:24.239824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33183 (local address 127.0.0.1:42565)
0928 13:02:24.240074 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:24.240084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:24.240120 (+ 36us) server_negotiation.cc:407] Connection header received
0928 13:02:24.241447 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:24.241457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:24.241526 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:24.241682 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:24.242402 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:24.243906 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:24.245162 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:24.245716 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:24.245843 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:24.248525 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:24.248530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:24.248530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:24.248543 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:24.250322 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:24.250726 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:24.250729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:24.250730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:24.250827 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:24.251063 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:24.251064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:24.251064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:24.251322 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:24.251503 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:24.251600 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 13:02:24.251773 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":109,"threads_started":1}
I20260928 13:02:24.253052 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:24.257871 31936 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:24.258565 31936 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:24.262710 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:24.254492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54009 (local address 127.0.0.1:42565)
0928 13:02:24.254534 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:24.254536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:24.256229 (+ 1693us) server_negotiation.cc:407] Connection header received
0928 13:02:24.256309 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:24.256310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:24.256321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:24.256376 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:24.256935 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:24.257511 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:24.258389 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:24.258636 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:24.258698 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:24.260089 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:24.260090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:24.260090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:24.260097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:24.261539 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:24.261918 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:24.261919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:24.261919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:24.261978 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:24.262183 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:24.262183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:24.262184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:24.262305 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:24.262410 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:24.262514 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 13:02:24.262603 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:24.269502 31936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b54afe346b426cbab569eacd0d1860: Generated new TSK 0
I20260928 13:02:24.269729 31936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:24.272168 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31854
2026-09-28T13:02:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31953](info): set up 2 sockets
Sep 28 13:02:24 dist-test-slave-8sjz krb5kdc[31953](info): commencing operation
krb5kdc: starting...
W20260928 13:02:25.357249 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:25 dist-test-slave-8sjz krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:25.478969 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:25.562454 31966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:25.562716 31966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:25.562757 31966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:25.563961 31966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:25.564036 31966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:25.564091 31966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:25.564124 31966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:25.564149 31966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:25.565704 31966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:25.566025 31966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:25.566274 31966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:25.571398 31974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:25.571398 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:25.571409 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:25.571707 31966 server_base.cc:1034] running on GCE node
I20260928 13:02:25.571943 31966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:25.572288 31966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:25.573437 31966 hybrid_clock.cc:648] HybridClock initialized: now 1790600545573413 us; error 36 us; skew 500 ppm
Sep 28 13:02:25 dist-test-slave-8sjz krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:25.575959 31966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:25.576444 31966 webserver.cc:460] Webserver started at http://127.0.0.1:36187/ using document root <none> and password file <none>
I20260928 13:02:25.576660 31966 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:25.576705 31966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:25.576797 31966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:25.577865 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "372b5429c87b4684bce8a794475f04d2"
format_stamp: "Formatted at 2026-09-28 13:02:25 on dist-test-slave-8sjz"
server_key: "30d4f5fdd165ac2df404e34dabebf2b4"
server_key_iv: "6d30c2ffd411fb76d717220902ef51b6"
server_key_version: "encryptionkey@0"
I20260928 13:02:25.578176 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "372b5429c87b4684bce8a794475f04d2"
format_stamp: "Formatted at 2026-09-28 13:02:25 on dist-test-slave-8sjz"
server_key: "30d4f5fdd165ac2df404e34dabebf2b4"
server_key_iv: "6d30c2ffd411fb76d717220902ef51b6"
server_key_version: "encryptionkey@0"
I20260928 13:02:25.579512 31966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 13:02:25.580389 31981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:25.580570 31966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:25.580658 31966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "372b5429c87b4684bce8a794475f04d2"
format_stamp: "Formatted at 2026-09-28 13:02:25 on dist-test-slave-8sjz"
server_key: "30d4f5fdd165ac2df404e34dabebf2b4"
server_key_iv: "6d30c2ffd411fb76d717220902ef51b6"
server_key_version: "encryptionkey@0"
I20260928 13:02:25.580757 31966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:25.593839 31966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:25.596019 31966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:25.596263 31966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:25.600395 31966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41819
I20260928 13:02:25.600430 32033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41819 every 8 connection(s)
I20260928 13:02:25.600811 31966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:25.601749 32034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:25.604317 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2: Bootstrap starting.
I20260928 13:02:25.604374 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 31966
I20260928 13:02:25.604519 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:25.604744 11309 external_mini_cluster.cc:1428] Setting key 1afedfd7fb4f8607de2ec96781c1d89e
I20260928 13:02:25.605134 32034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:25.605574 32034 log.cc:824] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:25.606400 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2: No bootstrap required, opened a new log
I20260928 13:02:25.608704 32034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260928 13:02:25.608886 32034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:25.608932 32034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372b5429c87b4684bce8a794475f04d2, State: Initialized, Role: FOLLOWER
I20260928 13:02:25.609071 32034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260928 13:02:25.609167 32034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:25.609212 32034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:25.609261 32034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:25.609858 32034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260928 13:02:25.609982 32034 leader_election.cc:302] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372b5429c87b4684bce8a794475f04d2; no voters:
I20260928 13:02:25.610175 32034 leader_election.cc:288] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:25.610211 32039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:25.610383 32039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [term 1 LEADER]: Becoming Leader. State: Replica: 372b5429c87b4684bce8a794475f04d2, State: Running, Role: LEADER
I20260928 13:02:25.610443 32034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:25.610514 32039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260928 13:02:25.610896 32039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372b5429c87b4684bce8a794475f04d2. Latest consensus state: current_term: 1 leader_uuid: "372b5429c87b4684bce8a794475f04d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260928 13:02:25.610921 32041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372b5429c87b4684bce8a794475f04d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b5429c87b4684bce8a794475f04d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260928 13:02:25.611104 32041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:25.611014 32039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:25.610956 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:25.605300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50747 (local address 127.0.0.1:41819)
0928 13:02:25.605495 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:25.605504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:25.605531 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:02:25.606165 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:25.606173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:25.606225 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:25.606359 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:25.606588 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:25.607459 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:25.608668 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:25.609166 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:25.609298 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:25.610048 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:25.610053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:25.610053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:25.610292 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:25.610342 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:25.610358 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:25.610613 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:25.610648 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 13:02:25.610727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260928 13:02:25.611449 32048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:25.612177 32048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:25.612396 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:25.613940 32048 catalog_manager.cc:1347] Generated new cluster ID: 76eb2928647b4043975be0da8fcde8b1
I20260928 13:02:25.613992 32048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:25.619148 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:25.613778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35335 (local address 127.0.0.1:41819)
0928 13:02:25.613811 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:25.613813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:25.616338 (+ 2525us) server_negotiation.cc:407] Connection header received
0928 13:02:25.616403 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:25.616404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:25.616414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:25.616465 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:25.616612 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:25.617108 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:25.617849 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:25.618058 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:25.618134 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:25.618877 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:25.618878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:25.618879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:25.618883 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:25.618899 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:25.618903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:25.618955 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:25.619021 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 13:02:25.619044 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 13:02:25.634516 32048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:25.635123 32048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:25.640738 32048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372b5429c87b4684bce8a794475f04d2: Generated new TSK 0
I20260928 13:02:25.640967 32048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:25.650030 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 31966
2026-09-28T13:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:25 dist-test-slave-8sjz krb5kdc[32065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:25 dist-test-slave-8sjz krb5kdc[32065](info): set up 2 sockets
Sep 28 13:02:25 dist-test-slave-8sjz krb5kdc[32065](info): commencing operation
krb5kdc: starting...
W20260928 13:02:26.689371 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:26 dist-test-slave-8sjz krb5kdc[32065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:26.810528 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:26.894747 32078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:26.895087 32078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:26.895135 32078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:26.897495 32078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:26.897584 32078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:26.897611 32078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:26.897635 32078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:26.897658 32078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:26.900437 32078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:26.900775 32078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:26.901046 32078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:26.906643 32083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:26.906675 32086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:26.906791 32078 server_base.cc:1034] running on GCE node
W20260928 13:02:26.906934 32084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:26.907192 32078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:26.907572 32078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:26.908723 32078 hybrid_clock.cc:648] HybridClock initialized: now 1790600546908712 us; error 34 us; skew 500 ppm
Sep 28 13:02:26 dist-test-slave-8sjz krb5kdc[32065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:26.915632 32078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:26.916162 32078 webserver.cc:460] Webserver started at http://127.0.0.1:33181/ using document root <none> and password file <none>
I20260928 13:02:26.916400 32078 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:26.916443 32078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:26.916538 32078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:26.917631 32078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "60318f925fd54a3599ac157142cc00f2"
format_stamp: "Formatted at 2026-09-28 13:02:26 on dist-test-slave-8sjz"
server_key: "0b7887ad660d1c3dfbd39d1e4bad3f31"
server_key_iv: "2b561b12d69cff675993086fb8897f81"
server_key_version: "encryptionkey@0"
I20260928 13:02:26.917935 32078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "60318f925fd54a3599ac157142cc00f2"
format_stamp: "Formatted at 2026-09-28 13:02:26 on dist-test-slave-8sjz"
server_key: "0b7887ad660d1c3dfbd39d1e4bad3f31"
server_key_iv: "2b561b12d69cff675993086fb8897f81"
server_key_version: "encryptionkey@0"
I20260928 13:02:26.919214 32078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:26.919962 32093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:26.920159 32078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:26.920248 32078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "60318f925fd54a3599ac157142cc00f2"
format_stamp: "Formatted at 2026-09-28 13:02:26 on dist-test-slave-8sjz"
server_key: "0b7887ad660d1c3dfbd39d1e4bad3f31"
server_key_iv: "2b561b12d69cff675993086fb8897f81"
server_key_version: "encryptionkey@0"
I20260928 13:02:26.920308 32078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:26.923614 32078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:26.925681 32078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:26.925788 32078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:26.929620 32078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45939
I20260928 13:02:26.929816 32145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45939 every 8 connection(s)
I20260928 13:02:26.930048 32078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:26.931169 32146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:26.933744 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: Bootstrap starting.
I20260928 13:02:26.934376 32146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:26.934681 32146 log.cc:824] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:26.935525 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: No bootstrap required, opened a new log
I20260928 13:02:26.935775 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32078
I20260928 13:02:26.935884 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:26.936112 11309 external_mini_cluster.cc:1428] Setting key 2152ad874c273617d1f9b7346187151b
I20260928 13:02:26.937786 32146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260928 13:02:26.937961 32146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:26.938007 32146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60318f925fd54a3599ac157142cc00f2, State: Initialized, Role: FOLLOWER
I20260928 13:02:26.938136 32146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260928 13:02:26.938243 32146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:26.938278 32146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:26.938328 32146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:26.939193 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:26.936693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45803 (local address 127.0.0.1:45939)
0928 13:02:26.936885 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:26.936896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:26.936932 (+ 36us) server_negotiation.cc:407] Connection header received
0928 13:02:26.937757 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:26.937784 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:26.937817 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:26.937980 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:26.938713 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:26.938717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:26.938717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:26.938724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:26.938789 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:26.938802 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:26.938882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:26.938953 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 13:02:26.939000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260928 13:02:26.939491 32146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260928 13:02:26.939604 32146 leader_election.cc:302] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60318f925fd54a3599ac157142cc00f2; no voters:
I20260928 13:02:26.939805 32146 leader_election.cc:288] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:26.939844 32151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:26.940001 32151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [term 1 LEADER]: Becoming Leader. State: Replica: 60318f925fd54a3599ac157142cc00f2, State: Running, Role: LEADER
I20260928 13:02:26.940104 32146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:26.940121 32151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } }
I20260928 13:02:26.940200 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:26.940505 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60318f925fd54a3599ac157142cc00f2. Latest consensus state: current_term: 1 leader_uuid: "60318f925fd54a3599ac157142cc00f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } } }
I20260928 13:02:26.940583 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [sys.catalog]: This master's current role is: LEADER
W20260928 13:02:26.941962 32170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:26.942035 32170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:26.942101 32158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:26.942198 32152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60318f925fd54a3599ac157142cc00f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60318f925fd54a3599ac157142cc00f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45939 } } }
I20260928 13:02:26.942261 32158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:26.943019 32152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:26.944280 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:26.941942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40991 (local address 127.0.0.1:45939)
0928 13:02:26.941981 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:26.941984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:26.943136 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 13:02:26.943265 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:26.943266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:26.943273 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:26.943334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:26.944011 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:26.944012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:26.944013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:26.944017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:26.944061 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:26.944067 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:26.944130 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:26.944193 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 13:02:26.944208 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:26.944329 32158 catalog_manager.cc:1347] Generated new cluster ID: 1cb07ca1c7d34313a27855542bc8d564
I20260928 13:02:26.944388 32158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:26.962342 32158 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:26.962563 32158 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:26.966630 32158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60318f925fd54a3599ac157142cc00f2: Generated new TSK 0
I20260928 13:02:26.966857 32158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:26.978417 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32078
2026-09-28T13:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:27 dist-test-slave-8sjz krb5kdc[32177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:27 dist-test-slave-8sjz krb5kdc[32177](info): set up 2 sockets
Sep 28 13:02:27 dist-test-slave-8sjz krb5kdc[32177](info): commencing operation
krb5kdc: starting...
W20260928 13:02:28.089283 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:28.215107 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:28.297772 32190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:28.298048 32190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:28.298087 32190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:28.299520 32190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:28.299620 32190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:28.299647 32190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:28.299677 32190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:28.299701 32190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:28.301321 32190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:28.301615 32190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:28.301875 32190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:28.307165 32196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:28.307170 32198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:28.307174 32195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:28.307416 32190 server_base.cc:1034] running on GCE node
I20260928 13:02:28.307693 32190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:28.308003 32190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:28.309218 32190 hybrid_clock.cc:648] HybridClock initialized: now 1790600548309204 us; error 27 us; skew 500 ppm
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:28.311757 32190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:28.312249 32190 webserver.cc:460] Webserver started at http://127.0.0.1:45649/ using document root <none> and password file <none>
I20260928 13:02:28.312444 32190 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:28.312486 32190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:28.312578 32190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:28.313625 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "ff4f39766b4544f6b68a3d3cb0c1c332"
format_stamp: "Formatted at 2026-09-28 13:02:28 on dist-test-slave-8sjz"
server_key: "bed4a29c25744a67fe36af3a3ff79000"
server_key_iv: "4e8106702810df4c4e6c07f52e86d668"
server_key_version: "encryptionkey@0"
I20260928 13:02:28.313941 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "ff4f39766b4544f6b68a3d3cb0c1c332"
format_stamp: "Formatted at 2026-09-28 13:02:28 on dist-test-slave-8sjz"
server_key: "bed4a29c25744a67fe36af3a3ff79000"
server_key_iv: "4e8106702810df4c4e6c07f52e86d668"
server_key_version: "encryptionkey@0"
I20260928 13:02:28.315225 32190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:02:28.315987 32205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:28.316180 32190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:28.316270 32190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "ff4f39766b4544f6b68a3d3cb0c1c332"
format_stamp: "Formatted at 2026-09-28 13:02:28 on dist-test-slave-8sjz"
server_key: "bed4a29c25744a67fe36af3a3ff79000"
server_key_iv: "4e8106702810df4c4e6c07f52e86d668"
server_key_version: "encryptionkey@0"
I20260928 13:02:28.316327 32190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:28.327562 32190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:28.329593 32190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:28.329717 32190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:28.333657 32190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34497
I20260928 13:02:28.333693 32257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34497 every 8 connection(s)
I20260928 13:02:28.334125 32190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:28.335665 32258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:28.338068 32258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332: Bootstrap starting.
I20260928 13:02:28.338676 32258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:28.338984 32258 log.cc:824] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:28.339730 32258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332: No bootstrap required, opened a new log
I20260928 13:02:28.340493 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32190
I20260928 13:02:28.340615 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:28.340826 11309 external_mini_cluster.cc:1428] Setting key 94fe88b60f5e604dd41c851015ddba2a
I20260928 13:02:28.342157 32258 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260928 13:02:28.342305 32258 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:28.342329 32258 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4f39766b4544f6b68a3d3cb0c1c332, State: Initialized, Role: FOLLOWER
I20260928 13:02:28.342470 32258 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260928 13:02:28.342540 32258 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:28.342592 32258 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:28.342638 32258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:28.343431 32258 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260928 13:02:28.343575 32258 leader_election.cc:302] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4f39766b4544f6b68a3d3cb0c1c332; no voters:
I20260928 13:02:28.343763 32258 leader_election.cc:288] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:28.344015 32264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:28.344118 32258 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:28.344369 32264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [term 1 LEADER]: Becoming Leader. State: Replica: ff4f39766b4544f6b68a3d3cb0c1c332, State: Running, Role: LEADER
I20260928 13:02:28.344578 32264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260928 13:02:28.344995 32264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4f39766b4544f6b68a3d3cb0c1c332. Latest consensus state: current_term: 1 leader_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260928 13:02:28.345032 32263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4f39766b4544f6b68a3d3cb0c1c332" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260928 13:02:28.345137 32264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:28.345152 32263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:28.345491 32275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:28.346063 32275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:28.347890 32275 catalog_manager.cc:1347] Generated new cluster ID: 84499604c373491480c80d7f6440728a
I20260928 13:02:28.347949 32275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:02:28.352803 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:28.341405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:37405 (local address 127.0.0.1:34497)
0928 13:02:28.341577 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:28.341585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:28.341610 (+ 25us) server_negotiation.cc:407] Connection header received
0928 13:02:28.342336 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:28.342346 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:28.342423 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:28.342612 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:28.343428 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:28.344933 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:28.346167 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:28.346766 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:28.346878 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:28.349429 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:28.349433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:28.349433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:28.349445 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:28.351198 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:28.351617 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:28.351620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:28.351621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:28.351714 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:28.351949 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:28.351950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:28.351950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:28.352254 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:28.352447 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:28.352536 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 13:02:28.352637 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260928 13:02:28.353829 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:28.360680 32275 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:28.361393 32275 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:28.361939 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:28.355264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:43337 (local address 127.0.0.1:34497)
0928 13:02:28.355302 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:28.355304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:28.355426 (+ 122us) server_negotiation.cc:407] Connection header received
0928 13:02:28.355501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:28.355503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:28.355513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:28.355568 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:28.356160 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:28.356695 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:28.357421 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:28.357663 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:28.357740 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:28.358979 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:28.358981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:28.358981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:28.358989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:28.360639 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:28.361054 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:28.361056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:28.361056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:28.361113 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:28.361359 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:28.361360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:28.361360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:28.361511 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:28.361612 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:28.361745 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 13:02:28.361809 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":19}
I20260928 13:02:28.365669 32275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff4f39766b4544f6b68a3d3cb0c1c332: Generated new TSK 0
I20260928 13:02:28.365890 32275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:28.369663 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32190
W20260928 13:02:28.374234 32175 connection.cc:535] client connection to 127.0.0.1:34497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34497): Connection reset by peer (error 104)
2026-09-28T13:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32289](info): set up 2 sockets
Sep 28 13:02:28 dist-test-slave-8sjz krb5kdc[32289](info): commencing operation
krb5kdc: starting...
W20260928 13:02:29.421132 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:29 dist-test-slave-8sjz krb5kdc[32289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:29.546615 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:29.629196 32302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:29.629423 32302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:29.629447 32302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:29.630640 32302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:29.630684 32302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:29.630698 32302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:29.630710 32302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:29.630722 32302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:29.632254 32302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:29.632524 32302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:29.632767 32302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:29.637564 32307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:29.637560 32308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:29.637557 32310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:29.637795 32302 server_base.cc:1034] running on GCE node
I20260928 13:02:29.638159 32302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:29.638470 32302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:29.639626 32302 hybrid_clock.cc:648] HybridClock initialized: now 1790600549639596 us; error 50 us; skew 500 ppm
Sep 28 13:02:29 dist-test-slave-8sjz krb5kdc[32289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:29.642383 32302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:29.642953 32302 webserver.cc:460] Webserver started at http://127.0.0.1:42799/ using document root <none> and password file <none>
I20260928 13:02:29.643214 32302 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:29.643294 32302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:29.643448 32302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:29.644649 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "bc242c66829d4e33876bf6100fcff888"
format_stamp: "Formatted at 2026-09-28 13:02:29 on dist-test-slave-8sjz"
server_key: "7dcd7e649cf09106eae649107856b1c9"
server_key_iv: "03ba1605ed1ffb6645893d4c63e120ac"
server_key_version: "encryptionkey@0"
I20260928 13:02:29.645013 32302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "bc242c66829d4e33876bf6100fcff888"
format_stamp: "Formatted at 2026-09-28 13:02:29 on dist-test-slave-8sjz"
server_key: "7dcd7e649cf09106eae649107856b1c9"
server_key_iv: "03ba1605ed1ffb6645893d4c63e120ac"
server_key_version: "encryptionkey@0"
I20260928 13:02:29.646371 32302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:02:29.647083 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:29.647264 32302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:29.647374 32302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "bc242c66829d4e33876bf6100fcff888"
format_stamp: "Formatted at 2026-09-28 13:02:29 on dist-test-slave-8sjz"
server_key: "7dcd7e649cf09106eae649107856b1c9"
server_key_iv: "03ba1605ed1ffb6645893d4c63e120ac"
server_key_version: "encryptionkey@0"
I20260928 13:02:29.647454 32302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:29.668249 32302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:29.670214 32302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:29.670401 32302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:29.674676 32302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40455
I20260928 13:02:29.674715 32369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40455 every 8 connection(s)
I20260928 13:02:29.675083 32302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:29.675978 32370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:29.678392 32370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888: Bootstrap starting.
I20260928 13:02:29.679013 32370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:29.679314 32370 log.cc:824] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:29.680130 32370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888: No bootstrap required, opened a new log
I20260928 13:02:29.681756 32370 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } }
I20260928 13:02:29.681946 32370 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:29.681991 32370 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc242c66829d4e33876bf6100fcff888, State: Initialized, Role: FOLLOWER
I20260928 13:02:29.682107 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32302
I20260928 13:02:29.682154 32370 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } }
I20260928 13:02:29.682220 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:29.682246 32370 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:29.682287 32370 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:29.682354 32370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:29.682454 11309 external_mini_cluster.cc:1428] Setting key 57e7544eb6dabb2cc0cc633a527c9be3
I20260928 13:02:29.683090 32370 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } }
I20260928 13:02:29.683211 32370 leader_election.cc:302] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc242c66829d4e33876bf6100fcff888; no voters:
I20260928 13:02:29.683395 32370 leader_election.cc:288] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:29.683425 32375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:29.683526 32375 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [term 1 LEADER]: Becoming Leader. State: Replica: bc242c66829d4e33876bf6100fcff888, State: Running, Role: LEADER
I20260928 13:02:29.683609 32375 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } }
I20260928 13:02:29.683686 32370 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:29.683993 32375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc242c66829d4e33876bf6100fcff888. Latest consensus state: current_term: 1 leader_uuid: "bc242c66829d4e33876bf6100fcff888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } } }
I20260928 13:02:29.684114 32375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:29.684003 32376 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc242c66829d4e33876bf6100fcff888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc242c66829d4e33876bf6100fcff888" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40455 } } }
I20260928 13:02:29.684281 32376 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:29.685634 32379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:29.686146 32379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:29.687837 32379 catalog_manager.cc:1347] Generated new cluster ID: e19e52620b7f4ab5a83942e7cc178921
I20260928 13:02:29.687889 32379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:29.693007 32379 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:29.693607 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:29.683022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:50811 (local address 127.0.0.1:40455)
0928 13:02:29.683224 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:29.683236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:29.683271 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:02:29.683400 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:29.683406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:29.683485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:29.683900 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:29.684016 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:29.685387 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:29.686539 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:29.687081 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:29.687243 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:29.692665 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:29.692670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:29.692670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:29.693046 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:29.693090 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:29.693106 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:29.693204 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:29.693316 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 13:02:29.693377 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260928 13:02:29.693858 32379 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:29.694589 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:29.699805 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:29.696151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:39813 (local address 127.0.0.1:40455)
0928 13:02:29.696227 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:29.696229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:29.696287 (+ 58us) server_negotiation.cc:407] Connection header received
0928 13:02:29.696370 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:29.696371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:29.696380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:29.696430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:29.696559 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:29.697326 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:29.698418 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:29.698730 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:29.698800 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:29.699554 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:29.699555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:29.699555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:29.699560 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:29.699575 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:29.699579 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:29.699637 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:29.699713 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 13:02:29.699726 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:02:29.701408 32379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc242c66829d4e33876bf6100fcff888: Generated new TSK 0
I20260928 13:02:29.701601 32379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:29.704510 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32302
W20260928 13:02:29.708763 32287 connection.cc:535] client connection to 127.0.0.1:40455 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40455): Connection reset by peer (error 104)
2026-09-28T13:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:29 dist-test-slave-8sjz krb5kdc[32401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:29 dist-test-slave-8sjz krb5kdc[32401](info): set up 2 sockets
Sep 28 13:02:29 dist-test-slave-8sjz krb5kdc[32401](info): commencing operation
krb5kdc: starting...
W20260928 13:02:30.777248 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:30 dist-test-slave-8sjz krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:30.901324 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:30.986271 32414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:30.986495 32414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:30.986519 32414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:30.987663 32414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:30.987703 32414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:30.987717 32414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:30.987730 32414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:30.987742 32414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:30.989224 32414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:30.989440 32414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:30.989614 32414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:30.994613 32422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:30.994613 32419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:30.994635 32420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:30.994642 32414 server_base.cc:1034] running on GCE node
I20260928 13:02:30.995095 32414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:30.995419 32414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:30.996583 32414 hybrid_clock.cc:648] HybridClock initialized: now 1790600550996568 us; error 28 us; skew 500 ppm
Sep 28 13:02:30 dist-test-slave-8sjz krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:30.999111 32414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:30.999526 32414 webserver.cc:460] Webserver started at http://127.0.0.1:36887/ using document root <none> and password file <none>
I20260928 13:02:30.999724 32414 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:30.999768 32414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:30.999858 32414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:31.000955 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "a1efaba9493d47399d0d06079d7e4f0d"
format_stamp: "Formatted at 2026-09-28 13:02:30 on dist-test-slave-8sjz"
server_key: "fe2f351df0b63b45af256541c39a9076"
server_key_iv: "13974019f34ef24add32625be2417457"
server_key_version: "encryptionkey@0"
I20260928 13:02:31.001264 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "a1efaba9493d47399d0d06079d7e4f0d"
format_stamp: "Formatted at 2026-09-28 13:02:30 on dist-test-slave-8sjz"
server_key: "fe2f351df0b63b45af256541c39a9076"
server_key_iv: "13974019f34ef24add32625be2417457"
server_key_version: "encryptionkey@0"
I20260928 13:02:31.002517 32414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:02:31.003232 32429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:31.003402 32414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:02:31.003496 32414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "a1efaba9493d47399d0d06079d7e4f0d"
format_stamp: "Formatted at 2026-09-28 13:02:30 on dist-test-slave-8sjz"
server_key: "fe2f351df0b63b45af256541c39a9076"
server_key_iv: "13974019f34ef24add32625be2417457"
server_key_version: "encryptionkey@0"
I20260928 13:02:31.003556 32414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:31.006850 32414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:31.008966 32414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:31.009070 32414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:31.012748 32414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35657
I20260928 13:02:31.012842 32481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35657 every 8 connection(s)
I20260928 13:02:31.013161 32414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:31.014286 32482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:31.016589 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32414
I20260928 13:02:31.016714 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:31.016935 11309 external_mini_cluster.cc:1428] Setting key d4051f37da9c116f850f4f6be9b0ba5c
I20260928 13:02:31.016968 32482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d: Bootstrap starting.
I20260928 13:02:31.017724 32482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:31.018026 32482 log.cc:824] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:31.018781 32482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d: No bootstrap required, opened a new log
I20260928 13:02:31.019567 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:31.017454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:49727 (local address 127.0.0.1:35657)
0928 13:02:31.017650 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:31.017658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:31.017684 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:02:31.018201 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:31.018209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:31.018239 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:31.018413 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:31.019122 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:31.019126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:31.019126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:31.019131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:31.019175 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:31.019189 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:31.019296 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:31.019327 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 13:02:31.019373 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 13:02:31.020634 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:31.021414 32482 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } }
I20260928 13:02:31.021620 32482 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:31.021653 32482 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1efaba9493d47399d0d06079d7e4f0d, State: Initialized, Role: FOLLOWER
I20260928 13:02:31.021807 32482 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } }
I20260928 13:02:31.021924 32482 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:31.021962 32482 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:31.022018 32482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:31.022871 32482 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } }
I20260928 13:02:31.022995 32482 leader_election.cc:302] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1efaba9493d47399d0d06079d7e4f0d; no voters:
I20260928 13:02:31.023180 32482 leader_election.cc:288] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:31.023240 32492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:31.023370 32492 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [term 1 LEADER]: Becoming Leader. State: Replica: a1efaba9493d47399d0d06079d7e4f0d, State: Running, Role: LEADER
I20260928 13:02:31.023471 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:31.022228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:43101 (local address 127.0.0.1:35657)
0928 13:02:31.022267 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:31.022269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:31.022394 (+ 125us) server_negotiation.cc:407] Connection header received
0928 13:02:31.022423 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:31.022424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:31.022428 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:31.022501 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:31.023081 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:31.023083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:31.023083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:31.023088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:31.023106 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:31.023111 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:31.023181 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:31.023284 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 13:02:31.023328 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:02:31.023451 32482 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:31.023461 32492 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } }
I20260928 13:02:31.024109 32494 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1efaba9493d47399d0d06079d7e4f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } } }
I20260928 13:02:31.024228 32494 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:31.024631 32493 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1efaba9493d47399d0d06079d7e4f0d. Latest consensus state: current_term: 1 leader_uuid: "a1efaba9493d47399d0d06079d7e4f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1efaba9493d47399d0d06079d7e4f0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35657 } } }
I20260928 13:02:31.024718 32493 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:31.028095 32500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:31.028718 32500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:31.031112 32500 catalog_manager.cc:1347] Generated new cluster ID: c2e0f9add97b4470a2a77628d9524554
I20260928 13:02:31.031168 32500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:31.053709 32500 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:31.053938 32500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:31.068308 32500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1efaba9493d47399d0d06079d7e4f0d: Generated new TSK 0
I20260928 13:02:31.068557 32500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:31.072777 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32414
2026-09-28T13:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:31 dist-test-slave-8sjz krb5kdc[32513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:31 dist-test-slave-8sjz krb5kdc[32513](info): set up 2 sockets
Sep 28 13:02:31 dist-test-slave-8sjz krb5kdc[32513](info): commencing operation
krb5kdc: starting...
W20260928 13:02:32.105350 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:32.231412 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:32.313382 32526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:32.313607 32526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:32.313632 32526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:32.314925 32526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:32.314983 32526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:32.314998 32526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:32.315060 32526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:32.315084 32526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:32.316814 32526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:32.317041 32526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:32.317233 32526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:32.322319 32531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:32.322315 32534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:32.322288 32532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:32.322644 32526 server_base.cc:1034] running on GCE node
I20260928 13:02:32.322975 32526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:32.323246 32526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:32.324429 32526 hybrid_clock.cc:648] HybridClock initialized: now 1790600552324415 us; error 30 us; skew 500 ppm
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:32.333670 32526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:32.334267 32526 webserver.cc:460] Webserver started at http://127.0.0.1:35581/ using document root <none> and password file <none>
I20260928 13:02:32.334508 32526 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:32.334556 32526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:32.334659 32526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:32.335845 32526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "61dba2a19fd14d7e87b8f061a49fabcf"
format_stamp: "Formatted at 2026-09-28 13:02:32 on dist-test-slave-8sjz"
server_key: "7d08776ee83b9dcabbbeedecea8e1979"
server_key_iv: "e780ed285a15e6bf931a70456563f29e"
server_key_version: "encryptionkey@0"
I20260928 13:02:32.336283 32526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "61dba2a19fd14d7e87b8f061a49fabcf"
format_stamp: "Formatted at 2026-09-28 13:02:32 on dist-test-slave-8sjz"
server_key: "7d08776ee83b9dcabbbeedecea8e1979"
server_key_iv: "e780ed285a15e6bf931a70456563f29e"
server_key_version: "encryptionkey@0"
I20260928 13:02:32.337816 32526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:32.338826 32541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:32.339054 32526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:32.339144 32526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "61dba2a19fd14d7e87b8f061a49fabcf"
format_stamp: "Formatted at 2026-09-28 13:02:32 on dist-test-slave-8sjz"
server_key: "7d08776ee83b9dcabbbeedecea8e1979"
server_key_iv: "e780ed285a15e6bf931a70456563f29e"
server_key_version: "encryptionkey@0"
I20260928 13:02:32.339196 32526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:32.354656 32526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:32.357045 32526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:32.357260 32526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:32.361532 32526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41721
I20260928 13:02:32.361565 32593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41721 every 8 connection(s)
I20260928 13:02:32.361970 32526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:32.362946 32594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:32.365408 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf: Bootstrap starting.
I20260928 13:02:32.366008 32594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:32.366304 32594 log.cc:824] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:32.366693 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32526
I20260928 13:02:32.366808 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:32.366971 11309 external_mini_cluster.cc:1428] Setting key 57225d44c211b7e09194c7c6c0a43353
I20260928 13:02:32.367821 32594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf: No bootstrap required, opened a new log
I20260928 13:02:32.370961 32594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } }
I20260928 13:02:32.371173 32594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:32.371215 32594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61dba2a19fd14d7e87b8f061a49fabcf, State: Initialized, Role: FOLLOWER
I20260928 13:02:32.371351 32594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } }
I20260928 13:02:32.371428 32594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:32.371454 32594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:32.371497 32594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:32.372279 32594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } }
I20260928 13:02:32.372409 32594 leader_election.cc:302] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61dba2a19fd14d7e87b8f061a49fabcf; no voters:
I20260928 13:02:32.372615 32594 leader_election.cc:288] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:32.372650 32599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:32.372861 32599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [term 1 LEADER]: Becoming Leader. State: Replica: 61dba2a19fd14d7e87b8f061a49fabcf, State: Running, Role: LEADER
I20260928 13:02:32.372939 32594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:32.373024 32599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } }
I20260928 13:02:32.373517 32599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61dba2a19fd14d7e87b8f061a49fabcf. Latest consensus state: current_term: 1 leader_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } } }
I20260928 13:02:32.373664 32599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:32.373703 32601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61dba2a19fd14d7e87b8f061a49fabcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41721 } } }
I20260928 13:02:32.373799 32601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:32.373958 32607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:32.374475 32607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:02:32.376894 32607 catalog_manager.cc:1347] Generated new cluster ID: 377e618d7a86472cbf68d3434b1ea895
I20260928 13:02:32.376952 32607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:32.379498 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:32.367500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:37695 (local address 127.0.0.1:41721)
0928 13:02:32.367672 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:32.367680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:32.367706 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:02:32.368243 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:32.368252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:32.368307 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:32.368430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:32.369239 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:32.370726 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:32.371824 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:32.372318 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:32.372447 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:32.375604 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:32.375608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:32.375609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:32.375624 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:32.377660 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:32.378176 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:32.378179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:32.378179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:32.378285 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:32.378546 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:32.378546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:32.378547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:32.378839 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:32.378981 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:32.379179 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 13:02:32.379275 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 13:02:32.380638 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:32.385281 32607 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:32.386010 32607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:32.392493 32607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61dba2a19fd14d7e87b8f061a49fabcf: Generated new TSK 0
I20260928 13:02:32.392659 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:32.384452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:57229 (local address 127.0.0.1:41721)
0928 13:02:32.384493 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:32.384495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:32.385453 (+ 958us) server_negotiation.cc:407] Connection header received
0928 13:02:32.385566 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:32.385567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:32.385580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:32.385662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:32.386319 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:32.386909 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:32.387737 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:32.387988 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:32.388106 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:32.389505 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:32.389507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:32.389508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:32.389516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:32.391164 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:32.391576 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:32.391578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:32.391578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:32.391643 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:32.391906 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:32.391907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:32.391908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:32.392071 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:32.392186 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:32.392491 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 13:02:32.392570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:02:32.392791 32607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:32.395332 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32526
W20260928 13:02:32.400185 32511 connection.cc:535] client connection to 127.0.0.1:41721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41721): Connection reset by peer (error 104)
2026-09-28T13:02:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32625](info): set up 2 sockets
Sep 28 13:02:32 dist-test-slave-8sjz krb5kdc[32625](info): commencing operation
krb5kdc: starting...
W20260928 13:02:33.437335 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:33 dist-test-slave-8sjz krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:33.559108 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:33.641947 32638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:33.642218 32638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:33.642256 32638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:33.643471 32638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:33.643546 32638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:33.643570 32638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:33.643591 32638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:33.643621 32638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:33.645257 32638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:33.645593 32638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:33.645846 32638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:02:33.651113 32638 server_base.cc:1034] running on GCE node
W20260928 13:02:33.651077 32643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:33.651086 32644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:33.651059 32646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:33.651602 32638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:33.651867 32638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:33.653016 32638 hybrid_clock.cc:648] HybridClock initialized: now 1790600553652993 us; error 38 us; skew 500 ppm
Sep 28 13:02:33 dist-test-slave-8sjz krb5kdc[32625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:33.655614 32638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:33.656147 32638 webserver.cc:460] Webserver started at http://127.0.0.1:36129/ using document root <none> and password file <none>
I20260928 13:02:33.656415 32638 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:33.656463 32638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:33.656630 32638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:33.657745 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "a793a36af2b54c02aac0eb31b66db8c4"
format_stamp: "Formatted at 2026-09-28 13:02:33 on dist-test-slave-8sjz"
server_key: "413851a4724202aa1dd7d09d2fccd7a0"
server_key_iv: "36bf2752fe85f9168b5465b938b87a82"
server_key_version: "encryptionkey@0"
I20260928 13:02:33.658124 32638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "a793a36af2b54c02aac0eb31b66db8c4"
format_stamp: "Formatted at 2026-09-28 13:02:33 on dist-test-slave-8sjz"
server_key: "413851a4724202aa1dd7d09d2fccd7a0"
server_key_iv: "36bf2752fe85f9168b5465b938b87a82"
server_key_version: "encryptionkey@0"
I20260928 13:02:33.659488 32638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:33.660318 32653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:33.660499 32638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:33.660609 32638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "a793a36af2b54c02aac0eb31b66db8c4"
format_stamp: "Formatted at 2026-09-28 13:02:33 on dist-test-slave-8sjz"
server_key: "413851a4724202aa1dd7d09d2fccd7a0"
server_key_iv: "36bf2752fe85f9168b5465b938b87a82"
server_key_version: "encryptionkey@0"
I20260928 13:02:33.660686 32638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:33.676738 32638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:33.678755 32638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:33.678932 32638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:33.683127 32638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45601
I20260928 13:02:33.683166 32705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45601 every 8 connection(s)
I20260928 13:02:33.683540 32638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:33.684082 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32638
I20260928 13:02:33.684234 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:33.684449 11309 external_mini_cluster.cc:1428] Setting key 6b127b8e5868288037fdfab705e6fd8a
I20260928 13:02:33.684639 32706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:33.687781 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4: Bootstrap starting.
I20260928 13:02:33.688550 32706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:33.688861 32706 log.cc:824] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:33.689599 32706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4: No bootstrap required, opened a new log
I20260928 13:02:33.690963 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:33.685017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:51061 (local address 127.0.0.1:45601)
0928 13:02:33.685202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:33.685212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:33.685249 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:02:33.686131 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:33.686141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:33.686204 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:33.686354 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:33.686579 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:33.687585 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:33.688767 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:33.689269 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:33.689427 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:33.690266 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:33.690271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:33.690271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:33.690459 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:33.690491 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:33.690502 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:33.690606 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:33.690703 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 13:02:33.690741 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260928 13:02:33.691735 32706 raft_consensus.cc:348] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260928 13:02:33.691920 32706 raft_consensus.cc:374] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:33.691972 32706 raft_consensus.cc:729] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a793a36af2b54c02aac0eb31b66db8c4, State: Initialized, Role: FOLLOWER
I20260928 13:02:33.692101 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:33.692122 32706 consensus_queue.cc:260] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260928 13:02:33.692207 32706 raft_consensus.cc:388] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:33.692245 32706 raft_consensus.cc:482] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:33.692293 32706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:33.692847 32706 raft_consensus.cc:504] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260928 13:02:33.692972 32706 leader_election.cc:302] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a793a36af2b54c02aac0eb31b66db8c4; no voters:
I20260928 13:02:33.693141 32706 leader_election.cc:288] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:33.693167 32713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:33.693279 32713 raft_consensus.cc:686] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [term 1 LEADER]: Becoming Leader. State: Replica: a793a36af2b54c02aac0eb31b66db8c4, State: Running, Role: LEADER
I20260928 13:02:33.693414 32713 consensus_queue.cc:237] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } }
I20260928 13:02:33.693418 32706 sys_catalog.cc:564] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:33.693948 32713 sys_catalog.cc:455] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a793a36af2b54c02aac0eb31b66db8c4. Latest consensus state: current_term: 1 leader_uuid: "a793a36af2b54c02aac0eb31b66db8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } } }
I20260928 13:02:33.693987 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a793a36af2b54c02aac0eb31b66db8c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793a36af2b54c02aac0eb31b66db8c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45601 } } }
I20260928 13:02:33.694100 32713 sys_catalog.cc:458] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:33.694150 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:33.694520 32725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:33.695070 32725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:33.696902 32725 catalog_manager.cc:1347] Generated new cluster ID: be195bc1f25049629d7daab0ccf53cc2
I20260928 13:02:33.696957 32725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:33.697836 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:33.693639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:50019 (local address 127.0.0.1:45601)
0928 13:02:33.694347 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:33.694349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:33.694372 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:02:33.694394 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:33.694395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:33.694405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:33.694644 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:33.694693 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:33.695450 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:33.696445 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:33.696711 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:33.696748 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:33.697612 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:33.697613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:33.697613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:33.697617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:33.697632 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:33.697637 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:33.697691 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:33.697751 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 13:02:33.697760 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690}
I20260928 13:02:33.704291 32725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:33.704975 32725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:33.715629 32725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a793a36af2b54c02aac0eb31b66db8c4: Generated new TSK 0
I20260928 13:02:33.715842 32725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:33.721562 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32638
W20260928 13:02:33.726449 32623 connection.cc:535] client connection to 127.0.0.1:45601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45601): Connection reset by peer (error 104)
2026-09-28T13:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:33 dist-test-slave-8sjz krb5kdc[32737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:33 dist-test-slave-8sjz krb5kdc[32737](info): set up 2 sockets
Sep 28 13:02:33 dist-test-slave-8sjz krb5kdc[32737](info): commencing operation
krb5kdc: starting...
W20260928 13:02:34.761307 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:34 dist-test-slave-8sjz krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:02:34.885142 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:34.969297 32750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:34.969599 32750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:34.969642 32750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:34.970897 32750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:34.970945 32750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:34.970959 32750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:34.970973 32750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:34.970984 32750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:34.972571 32750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:34.972862 32750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:34.973120 32750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:34.978305 32756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:34.978296 32758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:34.978296 32755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:34.978392 32750 server_base.cc:1034] running on GCE node
I20260928 13:02:34.978792 32750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:34.979132 32750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:34.980295 32750 hybrid_clock.cc:648] HybridClock initialized: now 1790600554980266 us; error 54 us; skew 500 ppm
Sep 28 13:02:34 dist-test-slave-8sjz krb5kdc[32737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:34.982944 32750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:02:34.983449 32750 webserver.cc:460] Webserver started at http://127.0.0.1:45635/ using document root <none> and password file <none>
I20260928 13:02:34.983708 32750 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:34.983762 32750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:34.983911 32750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:34.985103 32750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "8487ac802f554d63b0fded2d10704e2b"
format_stamp: "Formatted at 2026-09-28 13:02:34 on dist-test-slave-8sjz"
server_key: "e0511ef63a4974fb3c9bc05a6adee2ed"
server_key_iv: "552b5d8949f69055dd2fc98049c5cf09"
server_key_version: "encryptionkey@0"
I20260928 13:02:34.985493 32750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "8487ac802f554d63b0fded2d10704e2b"
format_stamp: "Formatted at 2026-09-28 13:02:34 on dist-test-slave-8sjz"
server_key: "e0511ef63a4974fb3c9bc05a6adee2ed"
server_key_iv: "552b5d8949f69055dd2fc98049c5cf09"
server_key_version: "encryptionkey@0"
I20260928 13:02:34.986953 32750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:34.987839 32765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:34.988116 32750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:34.988246 32750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "8487ac802f554d63b0fded2d10704e2b"
format_stamp: "Formatted at 2026-09-28 13:02:34 on dist-test-slave-8sjz"
server_key: "e0511ef63a4974fb3c9bc05a6adee2ed"
server_key_iv: "552b5d8949f69055dd2fc98049c5cf09"
server_key_version: "encryptionkey@0"
I20260928 13:02:34.988332 32750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:34.991760 32750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:34.993907 32750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:34.994083 32750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:34.998024 32750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33041
I20260928 13:02:34.998055 349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33041 every 8 connection(s)
I20260928 13:02:34.998409 32750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:34.999372 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:35.000165 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 32750
I20260928 13:02:35.000291 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:35.000486 11309 external_mini_cluster.cc:1428] Setting key ca7b34dc10635ed116b1ea7040f4c8c7
I20260928 13:02:35.002590 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b: Bootstrap starting.
I20260928 13:02:35.003149 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:35.001046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:51051 (local address 127.0.0.1:33041)
0928 13:02:35.001238 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:35.001254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:35.001280 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:02:35.001722 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:35.001728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:35.001754 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:35.001938 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:35.002633 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:35.002638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:35.002638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:35.002643 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:35.002690 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:35.002704 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:35.002790 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:35.002861 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 13:02:35.003002 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 13:02:35.003207 350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:35.003546 350 log.cc:824] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:35.004279 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:35.004419 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b: No bootstrap required, opened a new log
I20260928 13:02:35.006515 350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260928 13:02:35.006682 350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:35.006734 350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8487ac802f554d63b0fded2d10704e2b, State: Initialized, Role: FOLLOWER
I20260928 13:02:35.006805 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:35.005726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.196:46617 (local address 127.0.0.1:33041)
0928 13:02:35.005771 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:35.005773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:35.005827 (+ 54us) server_negotiation.cc:407] Connection header received
0928 13:02:35.005896 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:35.005897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:35.005901 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:35.005955 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:35.006552 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:35.006553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:35.006554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:35.006558 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:35.006572 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:35.006576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:35.006632 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:35.006685 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 13:02:35.006698 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:02:35.006935 350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260928 13:02:35.007045 350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:35.007083 350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:35.007136 350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:35.007745 350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260928 13:02:35.007866 350 leader_election.cc:302] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8487ac802f554d63b0fded2d10704e2b; no voters:
I20260928 13:02:35.008021 350 leader_election.cc:288] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:35.008127 360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:35.008277 360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [term 1 LEADER]: Becoming Leader. State: Replica: 8487ac802f554d63b0fded2d10704e2b, State: Running, Role: LEADER
I20260928 13:02:35.008313 350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:35.008431 360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260928 13:02:35.008832 362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8487ac802f554d63b0fded2d10704e2b. Latest consensus state: current_term: 1 leader_uuid: "8487ac802f554d63b0fded2d10704e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } } }
I20260928 13:02:35.008955 362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:35.008822 361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8487ac802f554d63b0fded2d10704e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8487ac802f554d63b0fded2d10704e2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } } }
I20260928 13:02:35.009191 369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:35.009203 361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:35.009884 369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:35.011590 369 catalog_manager.cc:1347] Generated new cluster ID: 0131c8690a564f1996f8e816035eef2c
I20260928 13:02:35.011642 369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:35.030851 369 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:35.031154 369 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:35.057422 369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8487ac802f554d63b0fded2d10704e2b: Generated new TSK 0
I20260928 13:02:35.057660 369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:35.071759 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 32750
2026-09-28T13:02:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:35 dist-test-slave-8sjz krb5kdc[381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:35 dist-test-slave-8sjz krb5kdc[381](info): set up 2 sockets
Sep 28 13:02:35 dist-test-slave-8sjz krb5kdc[381](info): commencing operation
krb5kdc: starting...
W20260928 13:02:36.141198 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:36.267050 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43001
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:36.350878 394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:36.351166 394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:36.351209 394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:36.352510 394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:36.352600 394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:36.352625 394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:36.352646 394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:36.352679 394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:36.354290 394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:43001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:36.354617 394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:36.354879 394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:36.360322 400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:36.360364 394 server_base.cc:1034] running on GCE node
W20260928 13:02:36.360338 399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:36.360364 402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:36.360742 394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:36.361011 394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:36.362195 394 hybrid_clock.cc:648] HybridClock initialized: now 1790600556362162 us; error 53 us; skew 500 ppm
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:36.365056 394 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:36.365598 394 webserver.cc:460] Webserver started at http://127.11.11.126:37249/ using document root <none> and password file <none>
I20260928 13:02:36.365842 394 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:36.365914 394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:36.366077 394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:36.367184 394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "d3d909a3466c45b68308c8c814c98eb6"
format_stamp: "Formatted at 2026-09-28 13:02:36 on dist-test-slave-8sjz"
server_key: "ad8c0193cfbaaf1e1244bdfe9524a936"
server_key_iv: "50faf61c2a83111994c67e50534fd43e"
server_key_version: "encryptionkey@0"
I20260928 13:02:36.367575 394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "d3d909a3466c45b68308c8c814c98eb6"
format_stamp: "Formatted at 2026-09-28 13:02:36 on dist-test-slave-8sjz"
server_key: "ad8c0193cfbaaf1e1244bdfe9524a936"
server_key_iv: "50faf61c2a83111994c67e50534fd43e"
server_key_version: "encryptionkey@0"
I20260928 13:02:36.369200 394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:36.370046 410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:36.370255 394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:36.370373 394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "d3d909a3466c45b68308c8c814c98eb6"
format_stamp: "Formatted at 2026-09-28 13:02:36 on dist-test-slave-8sjz"
server_key: "ad8c0193cfbaaf1e1244bdfe9524a936"
server_key_iv: "50faf61c2a83111994c67e50534fd43e"
server_key_version: "encryptionkey@0"
I20260928 13:02:36.370451 394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:36.393749 394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:36.395715 394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:36.395905 394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:36.399932 394 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:43001
I20260928 13:02:36.399974 463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:43001 every 8 connection(s)
I20260928 13:02:36.400379 394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:36.401259 464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:36.402367 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 394
I20260928 13:02:36.402484 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:36.402678 11309 external_mini_cluster.cc:1428] Setting key 87a62bb9e5908534386e97d4bf0e831c
I20260928 13:02:36.403927 464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6: Bootstrap starting.
I20260928 13:02:36.404567 464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:36.404871 464 log.cc:824] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:36.405609 464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6: No bootstrap required, opened a new log
I20260928 13:02:36.407379 464 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } }
I20260928 13:02:36.407548 464 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:36.407589 464 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d909a3466c45b68308c8c814c98eb6, State: Initialized, Role: FOLLOWER
I20260928 13:02:36.407716 464 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } }
I20260928 13:02:36.407800 464 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:36.407836 464 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:36.407917 464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:36.408521 464 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } }
I20260928 13:02:36.408612 464 leader_election.cc:302] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d909a3466c45b68308c8c814c98eb6; no voters:
I20260928 13:02:36.408818 464 leader_election.cc:288] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:36.408869 469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:36.409101 469 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [term 1 LEADER]: Becoming Leader. State: Replica: d3d909a3466c45b68308c8c814c98eb6, State: Running, Role: LEADER
I20260928 13:02:36.409111 464 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:36.409238 469 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } }
I20260928 13:02:36.409533 471 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d909a3466c45b68308c8c814c98eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } } }
I20260928 13:02:36.409601 471 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [sys.catalog]: This master's current role is: LEADER
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:36.409819 475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:36.410351 475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:36.410718 470 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d909a3466c45b68308c8c814c98eb6. Latest consensus state: current_term: 1 leader_uuid: "d3d909a3466c45b68308c8c814c98eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d909a3466c45b68308c8c814c98eb6" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 43001 } } }
I20260928 13:02:36.411227 470 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:36.413272 475 catalog_manager.cc:1347] Generated new cluster ID: 3c1e2d798dbf40b48da3d58002703be0
I20260928 13:02:36.413365 475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:36.414507 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:36.403237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41357 (local address 127.11.11.126:43001)
0928 13:02:36.403424 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:36.403432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:36.403459 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:02:36.403883 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:36.403891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:36.403938 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:36.404126 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:36.404883 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:36.405930 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:36.406724 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:36.407144 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:36.407272 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:36.410278 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:36.410281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:36.410282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:36.410295 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:36.412496 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:36.412975 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:36.412978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:36.412978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:36.413087 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:36.413338 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:36.413339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:36.413339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:36.413644 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:36.413797 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:36.414203 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 13:02:36.414303 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260928 13:02:36.415607 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:36.424348 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:36.417015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54811 (local address 127.11.11.126:43001)
0928 13:02:36.417047 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:36.417049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:36.417161 (+ 112us) server_negotiation.cc:407] Connection header received
0928 13:02:36.417223 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:36.417223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:36.417232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:36.417283 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:36.417960 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:36.418710 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:36.419751 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:36.420090 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:36.420144 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:36.421591 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:36.421592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:36.421593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:36.421599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:36.423088 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:36.423470 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:36.423471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:36.423471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:36.423517 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:36.423726 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:36.423727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:36.423727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:36.423851 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:36.423931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:36.424194 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 13:02:36.424255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 13:02:36.431085 475 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:36.431645 475 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:36.439913 475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d909a3466c45b68308c8c814c98eb6: Generated new TSK 0
I20260928 13:02:36.440176 475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:36.448614 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 394
W20260928 13:02:36.453176 379 connection.cc:535] client connection to 127.11.11.126:43001 recv error: Network error: failed to read from TLS socket (remote: 127.11.11.126:43001): Connection reset by peer (error 104)
2026-09-28T13:02:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[495](info): set up 2 sockets
Sep 28 13:02:36 dist-test-slave-8sjz krb5kdc[495](info): commencing operation
krb5kdc: starting...
W20260928 13:02:37.493345 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:37 dist-test-slave-8sjz krb5kdc[495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:37.615513 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42051
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:37.699129 508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:37.699421 508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:37.699463 508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:37.700839 508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:37.700917 508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:37.700943 508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:37.700965 508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:37.700996 508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:37.702580 508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:42051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:37.702895 508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:37.703143 508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:37.708202 516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:37.708346 514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:37.708211 513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:37.708596 508 server_base.cc:1034] running on GCE node
I20260928 13:02:37.708865 508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:37.709146 508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:37.710321 508 hybrid_clock.cc:648] HybridClock initialized: now 1790600557710315 us; error 38 us; skew 500 ppm
Sep 28 13:02:37 dist-test-slave-8sjz krb5kdc[495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:37.725730 508 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:37.726290 508 webserver.cc:460] Webserver started at http://127.11.11.126:42077/ using document root <none> and password file <none>
I20260928 13:02:37.726570 508 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:37.726646 508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:37.726805 508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:37.727991 508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "360fd79c9cf641758179e802ca1c0805"
format_stamp: "Formatted at 2026-09-28 13:02:37 on dist-test-slave-8sjz"
server_key: "eda83fa65cedd1539c9e1fa90b58d032"
server_key_iv: "6dfc631cf0a97243c0302820fea54379"
server_key_version: "encryptionkey@0"
I20260928 13:02:37.728466 508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "360fd79c9cf641758179e802ca1c0805"
format_stamp: "Formatted at 2026-09-28 13:02:37 on dist-test-slave-8sjz"
server_key: "eda83fa65cedd1539c9e1fa90b58d032"
server_key_iv: "6dfc631cf0a97243c0302820fea54379"
server_key_version: "encryptionkey@0"
I20260928 13:02:37.729883 508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 13:02:37.730762 524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:37.730953 508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:37.731101 508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "360fd79c9cf641758179e802ca1c0805"
format_stamp: "Formatted at 2026-09-28 13:02:37 on dist-test-slave-8sjz"
server_key: "eda83fa65cedd1539c9e1fa90b58d032"
server_key_iv: "6dfc631cf0a97243c0302820fea54379"
server_key_version: "encryptionkey@0"
I20260928 13:02:37.731187 508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:37.747062 508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:37.749481 508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:37.749673 508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:37.754184 508 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:42051
I20260928 13:02:37.754226 576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:42051 every 8 connection(s)
I20260928 13:02:37.754602 508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:37.755513 577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:37.757881 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805: Bootstrap starting.
I20260928 13:02:37.758520 577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:37.758821 577 log.cc:824] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:37.759605 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805: No bootstrap required, opened a new log
I20260928 13:02:37.760890 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 508
I20260928 13:02:37.760993 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:37.761193 11309 external_mini_cluster.cc:1428] Setting key c782158c76c7fb79b6b435832172fa18
I20260928 13:02:37.761476 577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } }
I20260928 13:02:37.761701 577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:37.761781 577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360fd79c9cf641758179e802ca1c0805, State: Initialized, Role: FOLLOWER
I20260928 13:02:37.761962 577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } }
I20260928 13:02:37.762056 577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:37.762081 577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:37.762157 577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:37.762933 577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } }
I20260928 13:02:37.763028 577 leader_election.cc:302] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360fd79c9cf641758179e802ca1c0805; no voters:
I20260928 13:02:37.763275 577 leader_election.cc:288] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:37.763316 582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:37.763429 582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [term 1 LEADER]: Becoming Leader. State: Replica: 360fd79c9cf641758179e802ca1c0805, State: Running, Role: LEADER
I20260928 13:02:37.763582 577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:37.763569 582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } }
I20260928 13:02:37.764163 584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360fd79c9cf641758179e802ca1c0805. Latest consensus state: current_term: 1 leader_uuid: "360fd79c9cf641758179e802ca1c0805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } } }
I20260928 13:02:37.764278 584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:37.764494 583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360fd79c9cf641758179e802ca1c0805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360fd79c9cf641758179e802ca1c0805" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42051 } } }
I20260928 13:02:37.764833 583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:37.765057 591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:37.765605 591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:37.767258 591 catalog_manager.cc:1347] Generated new cluster ID: 36e4a485582342abb3014be4a072e3c2
I20260928 13:02:37.767313 591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:37.767675 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:37.761798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43593 (local address 127.11.11.126:42051)
0928 13:02:37.761982 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:37.761990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:37.762018 (+ 28us) server_negotiation.cc:407] Connection header received
0928 13:02:37.762209 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:37.762218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:37.762280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:37.762411 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:37.762771 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:37.764023 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:37.765301 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:37.765909 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:37.766027 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:37.766842 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:37.766847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:37.766847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:37.767103 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:37.767147 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:37.767163 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:37.767308 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:37.767430 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 13:02:37.767460 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 13:02:37.768802 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:37.773866 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:37.770222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34537 (local address 127.11.11.126:42051)
0928 13:02:37.770318 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:37.770320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:37.770372 (+ 52us) server_negotiation.cc:407] Connection header received
0928 13:02:37.770438 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:37.770439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:37.770449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:37.770509 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:37.770637 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:37.771392 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:37.772422 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:37.772750 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:37.772828 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:37.773591 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:37.773593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:37.773593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:37.773598 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:37.773616 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:37.773620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:37.773687 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:37.773759 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 13:02:37.773775 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 13:02:37.786070 591 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:37.787215 591 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:37.789754 591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360fd79c9cf641758179e802ca1c0805: Generated new TSK 0
I20260928 13:02:37.789980 591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:37.798851 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 508
W20260928 13:02:37.803229 493 connection.cc:535] client connection to 127.11.11.126:42051 recv error: Network error: failed to read from TLS socket (remote: 127.11.11.126:42051): Connection reset by peer (error 104)
2026-09-28T13:02:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:37 dist-test-slave-8sjz krb5kdc[608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:37 dist-test-slave-8sjz krb5kdc[608](info): set up 2 sockets
Sep 28 13:02:37 dist-test-slave-8sjz krb5kdc[608](info): commencing operation
krb5kdc: starting...
W20260928 13:02:38.881242 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:38 dist-test-slave-8sjz krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:39.007270 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45343
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:43749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:39.092229 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:39.092456 621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:39.092480 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:39.093616 621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:39.093652 621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:39.093665 621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:39.093678 621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:39.093690 621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:39.095256 621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:43749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:39.095469 621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:39.095651 621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:39.100538 627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:39.100523 626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:39.100522 629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:39.100859 621 server_base.cc:1034] running on GCE node
I20260928 13:02:39.101171 621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:39.101486 621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:39.102636 621 hybrid_clock.cc:648] HybridClock initialized: now 1790600559102614 us; error 38 us; skew 500 ppm
Sep 28 13:02:39 dist-test-slave-8sjz krb5kdc[608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600559, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:39.105191 621 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:39.105639 621 webserver.cc:460] Webserver started at http://127.11.11.126:41923/ using document root <none> and password file <none>
I20260928 13:02:39.105880 621 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:39.105979 621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:39.106135 621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:39.107213 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "65f2aabb39794ecbbe11fe071e3a8a01"
format_stamp: "Formatted at 2026-09-28 13:02:39 on dist-test-slave-8sjz"
server_key: "6071c57f8267d0d267d0b743ce983bfe"
server_key_iv: "ee8d00b752be15f1a668ad50f696a380"
server_key_version: "encryptionkey@0"
I20260928 13:02:39.107590 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "65f2aabb39794ecbbe11fe071e3a8a01"
format_stamp: "Formatted at 2026-09-28 13:02:39 on dist-test-slave-8sjz"
server_key: "6071c57f8267d0d267d0b743ce983bfe"
server_key_iv: "ee8d00b752be15f1a668ad50f696a380"
server_key_version: "encryptionkey@0"
I20260928 13:02:39.109066 621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:39.109786 636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:39.109959 621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:39.110064 621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "65f2aabb39794ecbbe11fe071e3a8a01"
format_stamp: "Formatted at 2026-09-28 13:02:39 on dist-test-slave-8sjz"
server_key: "6071c57f8267d0d267d0b743ce983bfe"
server_key_iv: "ee8d00b752be15f1a668ad50f696a380"
server_key_version: "encryptionkey@0"
I20260928 13:02:39.110139 621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:39.113560 621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:39.115453 621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:39.115607 621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:39.119649 621 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45343
I20260928 13:02:39.119683 689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45343 every 8 connection(s)
I20260928 13:02:39.120111 621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:39.121111 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:39.122310 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 621
I20260928 13:02:39.122402 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:39.122614 11309 external_mini_cluster.cc:1428] Setting key 4a5bef55a84dfaf84dfa9d69e4b211d4
I20260928 13:02:39.124020 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01: Bootstrap starting.
I20260928 13:02:39.124748 690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:39.125116 690 log.cc:824] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:39.125305 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:39.123149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46599 (local address 127.11.11.126:45343)
0928 13:02:39.123339 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:39.123349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:39.123390 (+ 41us) server_negotiation.cc:407] Connection header received
0928 13:02:39.123934 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:39.123943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:39.123978 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:39.124157 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:39.124854 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:39.124858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:39.124858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:39.124863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:39.124911 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:39.124928 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:39.125019 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:39.125077 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 13:02:39.125118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 13:02:39.125845 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01: No bootstrap required, opened a new log
I20260928 13:02:39.126255 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:39.128521 690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } }
I20260928 13:02:39.128705 690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:39.128750 690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f2aabb39794ecbbe11fe071e3a8a01, State: Initialized, Role: FOLLOWER
I20260928 13:02:39.128840 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:39.127765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39919 (local address 127.11.11.126:45343)
0928 13:02:39.127804 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:39.127806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:39.127902 (+ 96us) server_negotiation.cc:407] Connection header received
0928 13:02:39.127980 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:39.127980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:39.127984 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:39.128029 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:39.128635 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:39.128636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:39.128636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:39.128639 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:39.128651 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:39.128655 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:39.128695 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:39.128756 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 13:02:39.128769 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:02:39.128923 690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } }
I20260928 13:02:39.129012 690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:39.129035 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:39.129096 690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:39.129654 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } }
I20260928 13:02:39.129741 690 leader_election.cc:302] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f2aabb39794ecbbe11fe071e3a8a01; no voters:
I20260928 13:02:39.129926 690 leader_election.cc:288] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:39.130008 700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:39.130160 700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [term 1 LEADER]: Becoming Leader. State: Replica: 65f2aabb39794ecbbe11fe071e3a8a01, State: Running, Role: LEADER
I20260928 13:02:39.130184 690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:39.130244 700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } }
I20260928 13:02:39.130631 701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } } }
I20260928 13:02:39.130663 702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f2aabb39794ecbbe11fe071e3a8a01. Latest consensus state: current_term: 1 leader_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f2aabb39794ecbbe11fe071e3a8a01" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45343 } } }
I20260928 13:02:39.130776 701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:39.130788 702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:39.131269 709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:39.131757 709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:39.133256 709 catalog_manager.cc:1347] Generated new cluster ID: d2c419dd8b5f4047a63d091781aa33ab
I20260928 13:02:39.133306 709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:39.145869 709 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:39.146086 709 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:39.150094 709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f2aabb39794ecbbe11fe071e3a8a01: Generated new TSK 0
I20260928 13:02:39.150303 709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:39.157749 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 621
2026-09-28T13:02:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:39 dist-test-slave-8sjz krb5kdc[721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:39 dist-test-slave-8sjz krb5kdc[721](info): set up 2 sockets
Sep 28 13:02:39 dist-test-slave-8sjz krb5kdc[721](info): commencing operation
krb5kdc: starting...
W20260928 13:02:40.193176 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:40.316107 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45081
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:45081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:40.400492 734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:40.400756 734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:40.400799 734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:40.402070 734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:40.402150 734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:40.402175 734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:40.402208 734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:40.402235 734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:40.403837 734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:45081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:45081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:40.404176 734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:40.404439 734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:40.409485 742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:40.409485 739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:40.409528 740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:40.409565 734 server_base.cc:1034] running on GCE node
I20260928 13:02:40.409896 734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:40.410161 734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:40.411319 734 hybrid_clock.cc:648] HybridClock initialized: now 1790600560411307 us; error 33 us; skew 500 ppm
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600560, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:40.413964 734 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:40.414433 734 webserver.cc:460] Webserver started at http://127.11.11.126:39701/ using document root <none> and password file <none>
I20260928 13:02:40.414642 734 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:40.414683 734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:40.414772 734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:40.415850 734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "d89f0823c6a247458452cf8276d438a9"
format_stamp: "Formatted at 2026-09-28 13:02:40 on dist-test-slave-8sjz"
server_key: "c03d3b9dc7612a609f78f2c21d68b50b"
server_key_iv: "7e2f60179c877b0e572af4fd27fa90d1"
server_key_version: "encryptionkey@0"
I20260928 13:02:40.416244 734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "d89f0823c6a247458452cf8276d438a9"
format_stamp: "Formatted at 2026-09-28 13:02:40 on dist-test-slave-8sjz"
server_key: "c03d3b9dc7612a609f78f2c21d68b50b"
server_key_iv: "7e2f60179c877b0e572af4fd27fa90d1"
server_key_version: "encryptionkey@0"
I20260928 13:02:40.417541 734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:40.418231 749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:40.418406 734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:02:40.418488 734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "d89f0823c6a247458452cf8276d438a9"
format_stamp: "Formatted at 2026-09-28 13:02:40 on dist-test-slave-8sjz"
server_key: "c03d3b9dc7612a609f78f2c21d68b50b"
server_key_iv: "7e2f60179c877b0e572af4fd27fa90d1"
server_key_version: "encryptionkey@0"
I20260928 13:02:40.418543 734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:40.426067 734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:40.427834 734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:40.427951 734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:40.431802 734 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:45081
I20260928 13:02:40.431932 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:45081 every 8 connection(s)
I20260928 13:02:40.432294 734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:40.433244 803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:40.435611 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9: Bootstrap starting.
I20260928 13:02:40.436249 803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:40.436502 803 log.cc:824] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:40.437300 803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9: No bootstrap required, opened a new log
I20260928 13:02:40.438867 803 raft_consensus.cc:348] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } }
I20260928 13:02:40.439014 803 raft_consensus.cc:374] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:40.439037 803 raft_consensus.cc:729] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89f0823c6a247458452cf8276d438a9, State: Initialized, Role: FOLLOWER
I20260928 13:02:40.439149 803 consensus_queue.cc:260] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } }
I20260928 13:02:40.439199 803 raft_consensus.cc:388] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:40.439221 803 raft_consensus.cc:482] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:40.439247 803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:40.440088 803 raft_consensus.cc:504] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } }
I20260928 13:02:40.440174 803 leader_election.cc:302] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d89f0823c6a247458452cf8276d438a9; no voters:
I20260928 13:02:40.440318 803 leader_election.cc:288] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:40.440387 806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:40.440539 806 raft_consensus.cc:686] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [term 1 LEADER]: Becoming Leader. State: Replica: d89f0823c6a247458452cf8276d438a9, State: Running, Role: LEADER
I20260928 13:02:40.440601 803 sys_catalog.cc:564] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:40.440629 806 consensus_queue.cc:237] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } }
I20260928 13:02:40.441018 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d89f0823c6a247458452cf8276d438a9. Latest consensus state: current_term: 1 leader_uuid: "d89f0823c6a247458452cf8276d438a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } } }
I20260928 13:02:40.441007 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d89f0823c6a247458452cf8276d438a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d89f0823c6a247458452cf8276d438a9" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 45081 } } }
I20260928 13:02:40.441107 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:40.441113 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:40.441286 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 734
I20260928 13:02:40.441318 812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:40.441406 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:40.441632 11309 external_mini_cluster.cc:1428] Setting key ea1711b7ed4b004ab552d8e837429f21
I20260928 13:02:40.441835 812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:40.443600 812 catalog_manager.cc:1347] Generated new cluster ID: db3f51a7e2f0408ea00c5a341bf16019
I20260928 13:02:40.443653 812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:40.451261 812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:40.452000 812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:40.452934 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:40.442298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52625 (local address 127.11.11.126:45081)
0928 13:02:40.442444 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:40.442455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:40.443238 (+ 783us) server_negotiation.cc:407] Connection header received
0928 13:02:40.443473 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:40.443480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:40.443562 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:40.443739 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:40.444630 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:40.445575 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:40.446368 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:40.446810 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:40.446941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:40.449374 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:40.449380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:40.449381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:40.449397 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:40.451272 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:40.451627 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:40.451630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:40.451631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:40.451725 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:40.451936 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:40.451937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:40.451938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:40.452223 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:40.452418 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:40.452488 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 13:02:40.452745 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260928 13:02:40.453851 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:40.462634 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:40.456297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51133 (local address 127.11.11.126:45081)
0928 13:02:40.456338 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:40.456340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:40.456544 (+ 204us) server_negotiation.cc:407] Connection header received
0928 13:02:40.456616 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:40.456617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:40.456627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:40.456682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:40.457196 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:40.457695 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:40.458433 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:40.458654 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:40.458739 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:40.459963 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:40.459964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:40.459964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:40.459971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:40.461446 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:40.461821 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:40.461822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:40.461822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:40.461882 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:40.462085 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:40.462086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:40.462086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:40.462201 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:40.462298 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:40.462451 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 13:02:40.462533 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:40.479439 812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d89f0823c6a247458452cf8276d438a9: Generated new TSK 0
I20260928 13:02:40.479696 812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:40.487263 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 734
2026-09-28T13:02:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[834](info): set up 2 sockets
Sep 28 13:02:40 dist-test-slave-8sjz krb5kdc[834](info): commencing operation
krb5kdc: starting...
W20260928 13:02:41.521361 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:41 dist-test-slave-8sjz krb5kdc[834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:41.643483 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42587
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:42587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:41.724606 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:41.724843 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:41.724869 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:41.725998 847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:41.726040 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:41.726054 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:41.726068 847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:41.726079 847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:41.727545 847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:42587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:42587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:41.727807 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:41.728024 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:41.732832 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:41.732841 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:41.732841 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:41.733500 847 server_base.cc:1034] running on GCE node
I20260928 13:02:41.733781 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:41.734057 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:41.735248 847 hybrid_clock.cc:648] HybridClock initialized: now 1790600561735225 us; error 32 us; skew 500 ppm
Sep 28 13:02:41 dist-test-slave-8sjz krb5kdc[834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600561, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:41.737762 847 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:41.738330 847 webserver.cc:460] Webserver started at http://127.11.11.126:33013/ using document root <none> and password file <none>
I20260928 13:02:41.738580 847 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:41.738648 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:41.738794 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:41.740026 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "71e6747d55c244af9ce27126131c2e28"
format_stamp: "Formatted at 2026-09-28 13:02:41 on dist-test-slave-8sjz"
server_key: "a41904f4655273be62f1f9987620320d"
server_key_iv: "76619337fb83498d28673f000da76eb8"
server_key_version: "encryptionkey@0"
I20260928 13:02:41.740453 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "71e6747d55c244af9ce27126131c2e28"
format_stamp: "Formatted at 2026-09-28 13:02:41 on dist-test-slave-8sjz"
server_key: "a41904f4655273be62f1f9987620320d"
server_key_iv: "76619337fb83498d28673f000da76eb8"
server_key_version: "encryptionkey@0"
I20260928 13:02:41.741770 847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:41.742491 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:41.742668 847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:02:41.742774 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "71e6747d55c244af9ce27126131c2e28"
format_stamp: "Formatted at 2026-09-28 13:02:41 on dist-test-slave-8sjz"
server_key: "a41904f4655273be62f1f9987620320d"
server_key_iv: "76619337fb83498d28673f000da76eb8"
server_key_version: "encryptionkey@0"
I20260928 13:02:41.742877 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:41.766165 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:41.768639 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:41.768821 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:41.772864 847 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:42587
I20260928 13:02:41.772918 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:42587 every 8 connection(s)
I20260928 13:02:41.773299 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:41.774163 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:41.776566 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28: Bootstrap starting.
I20260928 13:02:41.777153 915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:41.777444 915 log.cc:824] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:41.778204 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28: No bootstrap required, opened a new log
I20260928 13:02:41.778753 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 847
I20260928 13:02:41.778869 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:41.779068 11309 external_mini_cluster.cc:1428] Setting key 8e332ede4f78599448dbd3b25c0a1827
I20260928 13:02:41.780925 915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } }
I20260928 13:02:41.781131 915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:41.781172 915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e6747d55c244af9ce27126131c2e28, State: Initialized, Role: FOLLOWER
I20260928 13:02:41.781350 915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } }
I20260928 13:02:41.781454 915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:41.781481 915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:41.781553 915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:41.782150 915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } }
I20260928 13:02:41.782234 915 leader_election.cc:302] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71e6747d55c244af9ce27126131c2e28; no voters:
I20260928 13:02:41.782442 915 leader_election.cc:288] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:41.782464 920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:41.782578 920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [term 1 LEADER]: Becoming Leader. State: Replica: 71e6747d55c244af9ce27126131c2e28, State: Running, Role: LEADER
I20260928 13:02:41.782720 915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:41.782716 920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } }
I20260928 13:02:41.783212 922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71e6747d55c244af9ce27126131c2e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } } }
I20260928 13:02:41.783463 922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:41.783380 921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e6747d55c244af9ce27126131c2e28. Latest consensus state: current_term: 1 leader_uuid: "71e6747d55c244af9ce27126131c2e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71e6747d55c244af9ce27126131c2e28" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 42587 } } }
I20260928 13:02:41.783900 929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:41.784508 929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:41.783922 921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:41.786382 929 catalog_manager.cc:1347] Generated new cluster ID: d036b65198ba4ad382180e0d78b7d2a3
I20260928 13:02:41.786418 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:41.779667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52109 (local address 127.11.11.126:42587)
0928 13:02:41.779850 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:41.779860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:41.779933 (+ 73us) server_negotiation.cc:407] Connection header received
0928 13:02:41.780881 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:41.780890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:41.780959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:41.781162 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:41.781405 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:41.782865 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:41.784054 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:41.784731 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:41.784877 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:41.785628 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:41.785632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:41.785632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:41.785865 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:41.785907 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:41.785921 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:41.786016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:41.786096 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 13:02:41.786179 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 13:02:41.786446 929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:41.787508 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:41.792372 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:41.788948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38525 (local address 127.11.11.126:42587)
0928 13:02:41.788981 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:41.788983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:41.789089 (+ 106us) server_negotiation.cc:407] Connection header received
0928 13:02:41.789145 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:41.789146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:41.789155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:41.789206 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:41.789346 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:41.790029 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:41.791050 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:41.791327 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:41.791387 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:41.792131 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:41.792133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:41.792133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:41.792137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:41.792152 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:41.792157 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:41.792205 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:41.792266 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 13:02:41.792289 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 13:02:41.815724 929 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:41.816359 929 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:41.822189 929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e6747d55c244af9ce27126131c2e28: Generated new TSK 0
I20260928 13:02:41.822438 929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:41.833848 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 847
2026-09-28T13:02:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:41 dist-test-slave-8sjz krb5kdc[946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:41 dist-test-slave-8sjz krb5kdc[946](info): set up 2 sockets
Sep 28 13:02:41 dist-test-slave-8sjz krb5kdc[946](info): commencing operation
krb5kdc: starting...
W20260928 13:02:42.869359 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:42 dist-test-slave-8sjz krb5kdc[946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:42.995390 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34877
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:43.082486 959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:43.082707 959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:43.082731 959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:43.083982 959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:43.084060 959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:43.084076 959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:43.084137 959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:43.084160 959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:43.085692 959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:43.085968 959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:43.086216 959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:43.091583 965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:43.091586 967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:43.091583 964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:43.091830 959 server_base.cc:1034] running on GCE node
I20260928 13:02:43.092192 959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:43.092475 959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:43.093636 959 hybrid_clock.cc:648] HybridClock initialized: now 1790600563093623 us; error 28 us; skew 500 ppm
Sep 28 13:02:43 dist-test-slave-8sjz krb5kdc[946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600563, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:43.104768 959 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:43.105342 959 webserver.cc:460] Webserver started at http://127.11.11.126:46809/ using document root <none> and password file <none>
I20260928 13:02:43.105612 959 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:43.105664 959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:43.105798 959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:43.106945 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "f90d419061ef4a65afc93404afb0276d"
format_stamp: "Formatted at 2026-09-28 13:02:43 on dist-test-slave-8sjz"
server_key: "ce44163ca6a309cd824f3354f6fa153f"
server_key_iv: "6c85cb38a5e279ca5c4dd70434a5eb13"
server_key_version: "encryptionkey@0"
I20260928 13:02:43.107311 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "f90d419061ef4a65afc93404afb0276d"
format_stamp: "Formatted at 2026-09-28 13:02:43 on dist-test-slave-8sjz"
server_key: "ce44163ca6a309cd824f3354f6fa153f"
server_key_iv: "6c85cb38a5e279ca5c4dd70434a5eb13"
server_key_version: "encryptionkey@0"
I20260928 13:02:43.108831 959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:43.109669 974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:43.109869 959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:43.109966 959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "f90d419061ef4a65afc93404afb0276d"
format_stamp: "Formatted at 2026-09-28 13:02:43 on dist-test-slave-8sjz"
server_key: "ce44163ca6a309cd824f3354f6fa153f"
server_key_iv: "6c85cb38a5e279ca5c4dd70434a5eb13"
server_key_version: "encryptionkey@0"
I20260928 13:02:43.110045 959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:43.113587 959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:43.115965 959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:43.116173 959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:43.120386 959 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:34877
I20260928 13:02:43.120420 1027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:34877 every 8 connection(s)
I20260928 13:02:43.120769 959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:43.121704 1028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:43.123970 1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d: Bootstrap starting.
I20260928 13:02:43.124562 1028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:43.124799 1028 log.cc:824] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:43.125399 1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d: No bootstrap required, opened a new log
I20260928 13:02:43.126920 1028 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } }
I20260928 13:02:43.127065 1028 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:02:43.127089 1028 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90d419061ef4a65afc93404afb0276d, State: Initialized, Role: FOLLOWER
I20260928 13:02:43.127189 1028 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } }
I20260928 13:02:43.127269 1028 raft_consensus.cc:388] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:02:43.127305 1028 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:02:43.127372 1028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:43.128269 1028 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } }
I20260928 13:02:43.128382 1028 leader_election.cc:302] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f90d419061ef4a65afc93404afb0276d; no voters:
I20260928 13:02:43.128557 1028 leader_election.cc:288] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:02:43.128607 1031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:43.128798 1031 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [term 1 LEADER]: Becoming Leader. State: Replica: f90d419061ef4a65afc93404afb0276d, State: Running, Role: LEADER
I20260928 13:02:43.128840 1028 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:43.128890 1031 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } }
I20260928 13:02:43.129297 1032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f90d419061ef4a65afc93404afb0276d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } } }
I20260928 13:02:43.129312 1033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90d419061ef4a65afc93404afb0276d. Latest consensus state: current_term: 1 leader_uuid: "f90d419061ef4a65afc93404afb0276d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f90d419061ef4a65afc93404afb0276d" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 34877 } } }
I20260928 13:02:43.129458 1032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:43.129467 1033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:43.129714 1040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:43.130324 1040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:43.130599 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 959
I20260928 13:02:43.130702 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:43.130851 11309 external_mini_cluster.cc:1428] Setting key e46e3c168c8923e7a865197edcd03f15
I20260928 13:02:43.132157 1040 catalog_manager.cc:1347] Generated new cluster ID: 39ee0188cf234c23bac50722bf7a5463
I20260928 13:02:43.132208 1040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:43.133320 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:43.131599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47077 (local address 127.11.11.126:34877)
0928 13:02:43.131764 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:43.131773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:43.131805 (+ 32us) server_negotiation.cc:407] Connection header received
0928 13:02:43.131913 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:43.131919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:43.131942 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:43.132143 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:43.132886 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:43.132889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:43.132889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:43.132894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:43.132942 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:43.132955 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:43.133025 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:43.133099 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 13:02:43.133141 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 13:02:43.134254 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:43.137164 1050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:43.135588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45771 (local address 127.11.11.126:34877)
0928 13:02:43.135623 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:43.135625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:43.136202 (+ 577us) server_negotiation.cc:407] Connection header received
0928 13:02:43.136275 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:43.136276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:43.136281 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:43.136329 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:43.136928 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:43.136929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:43.136929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:02:43.136933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:43.136946 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:02:43.136951 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:43.137008 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:43.137064 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 13:02:43.137080 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:43.140431 1040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:43.140592 1040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:43.145726 1040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f90d419061ef4a65afc93404afb0276d: Generated new TSK 0
I20260928 13:02:43.145920 1040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:43.153930 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 959
2026-09-28T13:02:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1319 ms)
[----------] 18 tests from AuthTokenIssuingTest (24226 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:43 dist-test-slave-8sjz krb5kdc[1059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:43 dist-test-slave-8sjz krb5kdc[1059](info): set up 2 sockets
Sep 28 13:02:43 dist-test-slave-8sjz krb5kdc[1059](info): commencing operation
krb5kdc: starting...
W20260928 13:02:44.189275 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:44.319049 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41883
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:44.400966 1072 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:44.401252 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:44.401284 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:44.401330 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:44.402547 1072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:44.402618 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:44.402642 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:44.402662 1072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:44.402693 1072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:44.404255 1072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:44.404547 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:44.404788 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:44.409763 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.409755 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.409757 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:44.410020 1072 server_base.cc:1034] running on GCE node
I20260928 13:02:44.410241 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:44.410538 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:44.411685 1072 hybrid_clock.cc:648] HybridClock initialized: now 1790600564411664 us; error 33 us; skew 500 ppm
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:44.414333 1072 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:44.414819 1072 webserver.cc:460] Webserver started at http://127.11.11.126:41389/ using document root <none> and password file <none>
I20260928 13:02:44.415020 1072 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:44.415062 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:44.415156 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:44.416296 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "789b5d2f5615452d85e6e829182d3c40"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "b9bcf94804461c17c7af894da87313d7"
server_key_iv: "78c3a98a73a7659313aaecd2851e157d"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.416610 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "789b5d2f5615452d85e6e829182d3c40"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "b9bcf94804461c17c7af894da87313d7"
server_key_iv: "78c3a98a73a7659313aaecd2851e157d"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.417871 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:44.418749 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:44.418946 1072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:44.419066 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "789b5d2f5615452d85e6e829182d3c40"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "b9bcf94804461c17c7af894da87313d7"
server_key_iv: "78c3a98a73a7659313aaecd2851e157d"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.419147 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:44.436805 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:44.438849 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:44.439028 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:44.443320 1072 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:41883
I20260928 13:02:44.443351 1139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:41883 every 8 connection(s)
I20260928 13:02:44.443722 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:44.444149 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1072
I20260928 13:02:44.444279 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:44.444494 11309 external_mini_cluster.cc:1428] Setting key 9396d3622e6c363ded85a367825939fd
I20260928 13:02:44.444881 1140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:44.446594 1140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.11.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.125@KRBTEST.COM" created.
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:44.459812 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.447370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.11.11.126:41883)
0928 13:02:44.447541 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.447550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.447677 (+ 127us) server_negotiation.cc:407] Connection header received
0928 13:02:44.448511 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.448513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.448532 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.448606 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.449430 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.450268 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.451641 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.451971 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.452111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.456389 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.456397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.456398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.456406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.458165 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.458636 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.458638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.458639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.458716 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.458973 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.458973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.458974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.459192 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.459312 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.459540 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.459636 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260928 13:02:44.459884 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.447298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41883 (local address 127.0.0.1:39414)
0928 13:02:44.447527 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.447577 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.448331 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.448647 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.448650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.449058 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.449307 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.449314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.450375 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.450376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.451549 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.451552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.452112 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.452113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.452189 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.452748 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.452759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.456289 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.458244 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.458251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.458254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.458549 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.458788 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.458788 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.458789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.458906 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.459275 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.459283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.459378 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.459495 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.459735 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260928 13:02:44.460759 1140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:44.461166 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.461017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42090)
0928 13:02:44.461075 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.461132 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 13:02:44.461333 1091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
I20260928 13:02:44.461787 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.461671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42100)
0928 13:02:44.461721 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.461757 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:02:44.461906 1140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:41219: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
I20260928 13:02:44.474956 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:41219
--webserver_interface=127.11.11.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:02:44.511196 1140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } attempt: 1
I20260928 13:02:44.511859 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.511671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42104)
0928 13:02:44.511752 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.511808 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 13:02:44.512326 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.512216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42112)
0928 13:02:44.512273 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.512301 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 13:02:44.512480 1140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:41219: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
W20260928 13:02:44.558667 1147 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:44.558955 1147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:44.559000 1147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:44.559038 1147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:44.560315 1147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:44.560384 1147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:44.560406 1147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:44.560458 1147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:44.560488 1147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:44.562053 1147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.11.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:44.562355 1147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:44.562614 1147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:02:44.567750 1147 server_base.cc:1034] running on GCE node
W20260928 13:02:44.567710 1155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.567721 1153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.567727 1152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:44.568218 1147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:44.568527 1147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:44.569691 1147 hybrid_clock.cc:648] HybridClock initialized: now 1790600564569679 us; error 45 us; skew 500 ppm
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:44.572254 1147 init.cc:377] Logged in from keytab as kudu/127.11.11.125@KRBTEST.COM (short username kudu)
I20260928 13:02:44.572716 1147 webserver.cc:460] Webserver started at http://127.11.11.125:37385/ using document root <none> and password file <none>
I20260928 13:02:44.572958 1147 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:44.573030 1147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:44.573186 1147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:44.574281 1147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data/instance:
uuid: "b4120b2d0047410486ef9432783370d4"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "cb1210655de17f91d95e9a3614c93312"
server_key_iv: "78647d20e6c73325db9c9d6ef4416283"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.574653 1147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal/instance:
uuid: "b4120b2d0047410486ef9432783370d4"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "cb1210655de17f91d95e9a3614c93312"
server_key_iv: "78647d20e6c73325db9c9d6ef4416283"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.575979 1147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:44.576768 1162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:44.576961 1147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:44.577064 1147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "b4120b2d0047410486ef9432783370d4"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "cb1210655de17f91d95e9a3614c93312"
server_key_iv: "78647d20e6c73325db9c9d6ef4416283"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.577152 1147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:44.580628 1140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } attempt: 2
I20260928 13:02:44.581066 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.580921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42126)
0928 13:02:44.580988 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.581030 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 13:02:44.581415 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.581333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42140)
0928 13:02:44.581373 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.581398 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 13:02:44.581540 1140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:41219: Network error: Client connection negotiation failed: client connection to 127.11.11.125:41219: connect: Connection refused (error 111)
I20260928 13:02:44.589351 1147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:44.591305 1147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:44.591475 1147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:44.595352 1147 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.125:41219
I20260928 13:02:44.595414 1215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.125:41219 every 8 connection(s)
I20260928 13:02:44.595788 1147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
I20260928 13:02:44.596838 1216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:44.598438 1216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:44.600121 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1147
I20260928 13:02:44.600246 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-1/wal/instance
I20260928 13:02:44.600440 11309 external_mini_cluster.cc:1428] Setting key e1383a4f77cb55bbf374b01c3ee31938
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.124@KRBTEST.COM; defaulting to no policy
I20260928 13:02:44.608500 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.599185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.11.11.126:41883)
0928 13:02:44.599247 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.599249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.599472 (+ 223us) server_negotiation.cc:407] Connection header received
0928 13:02:44.599938 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.599940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.599954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.600023 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.600824 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.601583 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.602593 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.602807 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.602885 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.605519 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.605520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.605521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.605527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.606971 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.607465 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.607466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.607466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.607518 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.607818 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.607818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.607818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.607921 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.607978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.608316 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.608425 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:02:44.608759 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.599159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41883 (local address 127.0.0.1:39416)
0928 13:02:44.599332 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.599369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.599825 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.600144 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.600147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.600479 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.600729 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.600733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.601680 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.601681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.602494 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.602496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.602921 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.602922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.603060 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.603606 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.603615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.605424 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.607060 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.607068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.607070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.607383 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.607584 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.607586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.607586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.607741 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.607988 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.607997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.608180 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.608306 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.608492 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260928 13:02:44.609373 1216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.124@KRBTEST.COM" created.
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:44.617623 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.609755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.11.11.125:41219)
0928 13:02:44.609931 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.609936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.609961 (+ 25us) server_negotiation.cc:407] Connection header received
0928 13:02:44.609981 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.609982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.609993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.610056 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.610292 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.610985 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.611945 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.612485 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.612564 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.614602 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.614608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.614609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.614614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.616265 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.616685 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.616688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.616689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.616743 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.616979 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.616980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.616980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.617122 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.617193 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.617354 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.617504 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260928 13:02:44.617630 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.609678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42154)
0928 13:02:44.609821 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.609831 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.609863 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.610068 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.610069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.610181 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.610231 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.610232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.611070 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.611070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.611865 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.611867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.612574 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.612575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.612633 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.613098 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.613102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.614521 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.616346 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.616347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.616348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.616593 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.616821 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.616822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.616822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.616906 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.617190 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.617192 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.617245 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.617329 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.617458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2}
I20260928 13:02:44.618515 1216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:44.618881 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.618745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59492)
0928 13:02:44.618788 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.618861 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 13:02:44.619019 1163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
I20260928 13:02:44.619407 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.619317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59496)
0928 13:02:44.619362 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.619385 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:02:44.619524 1216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:40303: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
I20260928 13:02:44.629130 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:40303
--webserver_interface=127.11.11.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:02:44.668778 1216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } attempt: 1
I20260928 13:02:44.669400 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.669232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59510)
0928 13:02:44.669317 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.669365 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 13:02:44.669862 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.669727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59522)
0928 13:02:44.669796 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.669834 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 13:02:44.669996 1216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:40303: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
I20260928 13:02:44.672711 1140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } attempt: 3
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:44.680846 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.673129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.11.11.125:41219)
0928 13:02:44.673198 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.673200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.673213 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:02:44.673277 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.673278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.673291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.673352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.673660 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.674424 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.675494 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.675817 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.675892 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.678021 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.678022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.678022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.678030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.679553 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.679911 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.679911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.679912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.679974 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.680231 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.680232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.680233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.680384 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.680478 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.680634 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.680739 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 13:02:44.680847 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.673036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42168)
0928 13:02:44.673128 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.673142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.673225 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.673366 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.673367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.673519 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.673585 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.673587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.674534 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.674535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.675391 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.675395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.675903 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.675905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.675983 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.676476 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.676480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.677930 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.679654 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.679655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.679655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.679830 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.680081 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.680082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.680082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.680160 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.680466 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.680467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.680527 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.680607 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.680744 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 13:02:44.681278 1140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:44.681638 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.681527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59536)
0928 13:02:44.681579 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.681606 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:02:44.682001 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.681911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59540)
0928 13:02:44.681956 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.681979 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:02:44.682103 1140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:40303: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
I20260928 13:02:44.713258 1140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } attempt: 1
I20260928 13:02:44.713845 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.713666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59550)
0928 13:02:44.713751 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.713802 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 13:02:44.714259 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.714153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59562)
0928 13:02:44.714204 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.714236 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:02:44.714367 1140 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:40303: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
W20260928 13:02:44.715761 1223 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:44.715975 1223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:44.716065 1223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:44.716120 1223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:44.717304 1223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:44.717348 1223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:44.717362 1223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:44.717420 1223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:44.717442 1223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:44.718950 1223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:41883,127.11.11.125:41219,127.11.11.124:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.11.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:44.719219 1223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:44.719482 1223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:44.724563 1231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.724540 1229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:44.724563 1228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:44.724974 1223 server_base.cc:1034] running on GCE node
I20260928 13:02:44.725258 1223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:44.725574 1223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:44.726758 1223 hybrid_clock.cc:648] HybridClock initialized: now 1790600564726752 us; error 43 us; skew 500 ppm
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:44.729429 1223 init.cc:377] Logged in from keytab as kudu/127.11.11.124@KRBTEST.COM (short username kudu)
I20260928 13:02:44.729923 1223 webserver.cc:460] Webserver started at http://127.11.11.124:44795/ using document root <none> and password file <none>
I20260928 13:02:44.730116 1223 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:44.730157 1223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:44.730248 1223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:44.731325 1223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data/instance:
uuid: "5131ab8ef944496ca1b906883b3e475f"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "00061bb95ed931ff93bfed007c8f8bd3"
server_key_iv: "490ae32f0145cf440de8e0ac59b6e208"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.731662 1223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal/instance:
uuid: "5131ab8ef944496ca1b906883b3e475f"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "00061bb95ed931ff93bfed007c8f8bd3"
server_key_iv: "490ae32f0145cf440de8e0ac59b6e208"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.733079 1223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:44.733801 1238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:44.733997 1223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:44.734081 1223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "5131ab8ef944496ca1b906883b3e475f"
format_stamp: "Formatted at 2026-09-28 13:02:44 on dist-test-slave-8sjz"
server_key: "00061bb95ed931ff93bfed007c8f8bd3"
server_key_iv: "490ae32f0145cf440de8e0ac59b6e208"
server_key_version: "encryptionkey@0"
I20260928 13:02:44.734138 1223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:44.738148 1216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } attempt: 2
I20260928 13:02:44.738584 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.738436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59566)
0928 13:02:44.738500 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.738545 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:02:44.738957 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.738862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59570)
0928 13:02:44.738909 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.738936 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:02:44.739084 1216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:40303: Network error: Client connection negotiation failed: client connection to 127.11.11.124:40303: connect: Connection refused (error 111)
I20260928 13:02:44.745968 1223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:44.747972 1223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:44.748169 1223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:44.752394 1223 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.124:40303
I20260928 13:02:44.752421 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.124:40303 every 8 connection(s)
I20260928 13:02:44.752802 1223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
I20260928 13:02:44.753788 1291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:44.754130 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1223
I20260928 13:02:44.754207 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600290352246-11309-0/minicluster-data/master-2/wal/instance
I20260928 13:02:44.754437 11309 external_mini_cluster.cc:1428] Setting key 2a2c319374f31bd5b995c72a56a5a1f9
I20260928 13:02:44.755632 1291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:02:44.764497 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.756317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.11.11.126:41883)
0928 13:02:44.756361 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.756363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.756544 (+ 181us) server_negotiation.cc:407] Connection header received
0928 13:02:44.756626 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.756627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.756639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.756718 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.757213 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.757672 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.758541 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.758743 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.758829 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.761068 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.761071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.761071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.761080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.762746 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.763287 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.763287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.763288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.763338 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.763618 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.763619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.763620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.763758 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.763855 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.764292 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.764390 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:44.764673 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.754949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.11.11.124:40303)
0928 13:02:44.755151 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.755162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.755195 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:02:44.755622 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.755628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.755674 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.755825 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.756668 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.757541 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.758323 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.758652 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.758772 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.761555 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.761563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.761563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.761570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.763334 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.763682 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.763686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.763686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.763754 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.763956 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.763956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.763957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.764166 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.764315 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.764397 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.764484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260928 13:02:44.764676 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.756265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:41883 (local address 127.0.0.1:39426)
0928 13:02:44.756432 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.756475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.756561 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.756739 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.756742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.757011 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.757109 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.757113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.757758 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.757759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.758439 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.758441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.758863 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.758864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.758934 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.759525 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.759533 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.760974 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.762826 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.762832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.762835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.763211 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.763420 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.763421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.763421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.763542 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.763923 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.763929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.764121 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.764246 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.764406 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260928 13:02:44.765357 1291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:44.772143 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.765614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:41219 (local address 127.0.0.1:42182)
0928 13:02:44.765664 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.765673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.765707 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.765813 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.765813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.765974 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.766041 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.766043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.766655 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.766656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.767150 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.767151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.767536 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.767537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.767606 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.768068 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.768074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.769321 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.771106 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.771107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.771107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.771266 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.771493 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.771494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.771494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.771543 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.771792 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.771792 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.771832 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.771912 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.772011 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:02:44.772143 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.765668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.11.11.125:41219)
0928 13:02:44.765709 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.765711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.765730 (+ 19us) server_negotiation.cc:407] Connection header received
0928 13:02:44.765760 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.765761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.765769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.765808 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.766117 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.766571 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.767245 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.767465 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.767541 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.769426 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.769427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.769428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.769434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.771016 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.771340 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.771341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.771342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.771414 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.771605 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.771606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.771607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.771733 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.771785 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.771945 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.772027 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:44.772810 1291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:44.772881 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.765567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.11.11.126:41883)
0928 13:02:44.765602 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.765604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.765613 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:02:44.765650 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.765651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.765661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.765707 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.766333 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.766781 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.767449 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.767654 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.767730 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.770069 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.770071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.770071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.770079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.771529 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.772058 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.772060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.772060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.772116 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.772319 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.772321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.772321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.772467 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.772681 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.772722 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.772796 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:44.779441 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.773052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59580)
0928 13:02:44.773106 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.773117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.773152 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.773294 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.773295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.773430 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.773464 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.773466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.774239 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.774240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.774767 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.774768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.775115 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.775116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.775170 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.775562 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.775566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.776815 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.778434 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.778435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.778435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.778590 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.778793 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.778794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.778794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.778847 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.779098 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.779099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.779135 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.779206 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.779305 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:02:44.779501 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.773100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.11.11.124:40303)
0928 13:02:44.773136 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.773138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.773146 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:02:44.773213 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.773214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.773224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.773276 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.773599 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.774163 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.774845 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.775046 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.775096 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.776898 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.776899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.776899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.776906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.778350 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.778673 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.778674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.778674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.778729 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.778919 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.778919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.778920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.779032 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.779107 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.779230 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.779375 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":18}
I20260928 13:02:44.780800 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.773552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.11.11.125:41219)
0928 13:02:44.773588 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.773590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.773597 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:02:44.773615 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.773616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.773626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.773684 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.774094 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.774583 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.775376 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.775667 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.775719 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.778123 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.778124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.778124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.778130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.779575 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.780102 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.780103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.780104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.780163 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.780329 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.780330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.780330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.780460 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.780614 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.780650 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.780719 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:44.781432 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: Bootstrap starting.
I20260928 13:02:44.782483 1291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:44.782539 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:44.782783 1291 log.cc:824] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:44.783617 1291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: No bootstrap required, opened a new log
I20260928 13:02:44.783478 1114 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "789b5d2f5615452d85e6e829182d3c40"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39432
I20260928 13:02:44.785813 1291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.785975 1291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5131ab8ef944496ca1b906883b3e475f, State: Initialized, Role: FOLLOWER
I20260928 13:02:44.786051 1291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.786324 1297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.786401 1291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:44.786434 1297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:02:44.788242 1308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:44.788300 1308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:44.789512 1140 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } attempt: 2
I20260928 13:02:44.788590 1223 master_runner.cc:186] Error getting master registration for 127.11.11.126:41883: OK, instance_id { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" instance_seqno: 1790600564442129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:02:44.791560 1223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:02:44.797360 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.789823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.11.11.124:40303)
0928 13:02:44.789867 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.789869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.789880 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:44.789967 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.789968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.789978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.790032 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.790373 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.791108 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.791984 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.792295 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.792377 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.794491 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.794493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.794493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.794500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.796141 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.796532 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.796532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.796533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.796590 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.796796 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.796797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.796797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.796959 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.797025 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.797191 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.797268 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:02:44.797362 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.789764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59582)
0928 13:02:44.789813 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.789826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.789866 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.790045 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.790047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.790234 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.790282 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.790284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.791186 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.791187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.791899 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.791902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.792398 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.792399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.792484 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.793002 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.793007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.794392 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.796221 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.796222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.796223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.796454 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.796659 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.796660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.796660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.796730 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.797032 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.797033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.797082 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.797158 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.797257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 13:02:44.798969 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: Bootstrap starting.
I20260928 13:02:44.799716 1140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:44.800027 1140 log.cc:824] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:44.800921 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: No bootstrap required, opened a new log
I20260928 13:02:44.801956 1140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.802101 1140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789b5d2f5615452d85e6e829182d3c40, State: Initialized, Role: FOLLOWER
I20260928 13:02:44.802196 1140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.802498 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.802556 1140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:44.802601 1114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 13:02:44.802603 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:44.802660 1114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:44.803575 1114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.803875 1114 leader_election.cc:288] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [CANDIDATE]: Term 1 election: Requested vote from peers b4120b2d0047410486ef9432783370d4 (127.11.11.125:41219), 5131ab8ef944496ca1b906883b3e475f (127.11.11.124:40303)
I20260928 13:02:44.804576 1265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "789b5d2f5615452d85e6e829182d3c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5131ab8ef944496ca1b906883b3e475f"
I20260928 13:02:44.804737 1265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [term 0 FOLLOWER]: Advancing to term 1
W20260928 13:02:44.804797 1322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:44.804858 1322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:44.805709 1265 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b5d2f5615452d85e6e829182d3c40 in term 1.
I20260928 13:02:44.806771 1088 leader_election.cc:302] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5131ab8ef944496ca1b906883b3e475f, 789b5d2f5615452d85e6e829182d3c40; no voters:
I20260928 13:02:44.806917 1311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:44.807056 1311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [term 1 LEADER]: Becoming Leader. State: Replica: 789b5d2f5615452d85e6e829182d3c40, State: Running, Role: LEADER
I20260928 13:02:44.807137 1311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.807547 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789b5d2f5615452d85e6e829182d3c40. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.807629 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:44.805279 1072 master_runner.cc:186] Error getting master registration for 127.11.11.125:41219: OK, instance_id { permanent_uuid: "b4120b2d0047410486ef9432783370d4" instance_seqno: 1790600564594300 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:02:44.807966 1072 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:02:44.814014 1332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:44.814260 1332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:44.816453 1265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [term 1 FOLLOWER]: Refusing update from remote peer 789b5d2f5615452d85e6e829182d3c40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:44.816830 1328 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:44.818558 1297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789b5d2f5615452d85e6e829182d3c40. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.818665 1297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:44.820482 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.820573 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:44.821187 1297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.821285 1297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:44.821524 1333 mvcc.cc:205] Tried to move back new op lower bound from 7334299913484279808 to 7334299913450676224. Current Snapshot: MvccSnapshot[applied={T|T < 7334299913484279808}]
I20260928 13:02:44.822849 1332 catalog_manager.cc:1347] Generated new cluster ID: 89115d94dd954d03b74e5ea060ba4aab
I20260928 13:02:44.823053 1332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:44.825119 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.807469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.11.11.126:41883)
0928 13:02:44.807506 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.807508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.808469 (+ 961us) server_negotiation.cc:407] Connection header received
0928 13:02:44.808695 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.808696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.808705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.808735 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.809522 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.810249 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.812966 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.813277 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.813349 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.821014 (+ 7665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.821016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.821016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.821024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.822696 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.823368 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.823369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.823369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.823428 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.823924 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.823925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.823926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.824107 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.824227 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.824911 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.825014 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:44.826000 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.806976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.11.11.125:41219)
0928 13:02:44.807026 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.807029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.808534 (+ 1505us) server_negotiation.cc:407] Connection header received
0928 13:02:44.808640 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.808641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.808655 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.808727 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.810224 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.811001 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.813011 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.813324 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.813367 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.819037 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.819039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.819039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.819048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.820908 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.823619 (+ 2711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.823620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.823621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.823679 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.824436 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.824437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.824437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.824583 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.824670 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.825803 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.825904 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:02:44.827075 1334 mvcc.cc:205] Tried to move back new op lower bound from 7334299913484279808 to 7334299913450676224. Current Snapshot: MvccSnapshot[applied={T|T < 7334299913484279808}]
I20260928 13:02:44.830220 1216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } attempt: 3
I20260928 13:02:44.834532 1332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:44.835273 1332 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 13:02:44 dist-test-slave-8sjz krb5kdc[1059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600564, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:02:44.836758 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.820280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.11.11.124:40303)
0928 13:02:44.820326 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.820328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.820627 (+ 299us) server_negotiation.cc:407] Connection header received
0928 13:02:44.820732 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.820733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.820744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.820775 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.821476 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.822272 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.823355 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.823669 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.823715 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.828206 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.828207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.828207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.828215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.829938 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.830990 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.830991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.830991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.831049 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.831258 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.831259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.831259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.831408 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.831481 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.836531 (+ 5050us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.836618 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:02:44.845016 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.830485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:40303 (local address 127.0.0.1:59608)
0928 13:02:44.830527 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:44.830540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:44.830592 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:44.830944 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:44.830961 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:44.831121 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:44.831176 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.831178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.832871 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.832872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.833719 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:44.833722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.834540 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.834541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:44.834602 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.835006 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:44.835011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:44.836557 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:44.838725 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.838726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.838727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.838963 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.844281 (+ 5318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:44.844282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:44.844282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:44.844368 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:44.844656 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:44.844658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:44.844707 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:02:44.844798 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 13:02:44.844918 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 13:02:44.845057 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:44.830519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.11.11.124:40303)
0928 13:02:44.830700 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.830702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.830709 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:02:44.830761 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.830762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.830772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.831267 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.831335 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.832725 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.833871 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:44.834412 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:44.834773 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:44.836644 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:44.836646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:44.836646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:44.836652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:44.838653 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.844087 (+ 5434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.844088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.844089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.844193 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:44.844435 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:44.844436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:44.844436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:44.844586 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:44.844655 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:44.844820 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 13:02:44.844916 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 13:02:44.846027 1332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40: Generated new TSK 0
I20260928 13:02:44.846283 1332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:02:44.846611 1216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: Bootstrap starting.
I20260928 13:02:44.847347 1216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:44.847687 1216 log.cc:824] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:44.849592 1216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: No bootstrap required, opened a new log
I20260928 13:02:44.851593 1216 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.851719 1216 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4120b2d0047410486ef9432783370d4, State: Initialized, Role: FOLLOWER
I20260928 13:02:44.851795 1216 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } }
I20260928 13:02:44.852133 1216 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:44.853053 1190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "789b5d2f5615452d85e6e829182d3c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4120b2d0047410486ef9432783370d4"
I20260928 13:02:44.853163 1190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:44.854049 1190 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789b5d2f5615452d85e6e829182d3c40 in term 1.
I20260928 13:02:44.854352 1189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [term 1 FOLLOWER]: Refusing update from remote peer 789b5d2f5615452d85e6e829182d3c40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:44.854631 1337 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:44.856494 1343 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.856659 1343 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:44.856751 1343 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789b5d2f5615452d85e6e829182d3c40. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.856849 1343 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:44.856917 1343 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:44.857007 1343 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:02:44.857993 1355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:44.858328 1355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:44.864724 1357 mvcc.cc:205] Tried to move back new op lower bound from 7334299913601703936 to 7334299913450676224. Current Snapshot: MvccSnapshot[applied={T|T < 7334299913484279808}]
W20260928 13:02:44.878804 1144 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.877441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.11.11.126:41883)
0928 13:02:44.877484 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.877486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.878055 (+ 569us) server_negotiation.cc:407] Connection header received
0928 13:02:44.878164 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.878166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.878177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.878210 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.878727 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39458: BlockingRecv error: recv got EOF from 127.0.0.1:39458 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 13:02:44.879822 1340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.877213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.11.11.124:40303)
0928 13:02:44.877275 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.877278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.879230 (+ 1952us) server_negotiation.cc:407] Connection header received
0928 13:02:44.879367 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.879369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.879381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.879413 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.879739 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59614: BlockingRecv error: recv got EOF from 127.0.0.1:59614 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 13:02:44.880782 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.877696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.11.11.125:41219)
0928 13:02:44.877738 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.877740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.880244 (+ 2504us) server_negotiation.cc:407] Connection header received
0928 13:02:44.880365 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.880365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.880377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.880408 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.880721 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42210: BlockingRecv error: recv got EOF from 127.0.0.1:42210 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 13:02:44.883270 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.882518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.11.11.125:41219)
0928 13:02:44.882552 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.882554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.882764 (+ 210us) server_negotiation.cc:407] Connection header received
0928 13:02:44.882867 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.882868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.882887 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.882916 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.883227 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42212: BlockingRecv error: recv got EOF from 127.0.0.1:42212 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 13:02:44.886332 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.885565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.11.11.125:41219)
0928 13:02:44.885597 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.885599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.885833 (+ 234us) server_negotiation.cc:407] Connection header received
0928 13:02:44.885924 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.885924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.885933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.885961 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.886289 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42228: BlockingRecv error: recv got EOF from 127.0.0.1:42228 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 13:02:44.893078 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.892326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.11.11.125:41219)
0928 13:02:44.892361 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.892362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.892598 (+ 236us) server_negotiation.cc:407] Connection header received
0928 13:02:44.892688 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.892689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.892699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.892725 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.893034 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42236: BlockingRecv error: recv got EOF from 127.0.0.1:42236 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 13:02:44.899070 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.898279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.11.11.125:41219)
0928 13:02:44.898315 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.898317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.898577 (+ 260us) server_negotiation.cc:407] Connection header received
0928 13:02:44.898674 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.898675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.898685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.898713 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.899023 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42252: BlockingRecv error: recv got EOF from 127.0.0.1:42252 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 13:02:44.909364 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.908397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.11.11.125:41219)
0928 13:02:44.908443 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.908445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.908751 (+ 306us) server_negotiation.cc:407] Connection header received
0928 13:02:44.908862 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.908863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.908875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.908906 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.909294 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42266: BlockingRecv error: recv got EOF from 127.0.0.1:42266 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:02:44.928014 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.927028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.11.11.125:41219)
0928 13:02:44.927078 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.927081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.927354 (+ 273us) server_negotiation.cc:407] Connection header received
0928 13:02:44.927494 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.927495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.927509 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.927542 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.927952 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42276: BlockingRecv error: recv got EOF from 127.0.0.1:42276 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:02:44.962714 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:44.961719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.11.11.125:41219)
0928 13:02:44.961774 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:44.961777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:44.962100 (+ 323us) server_negotiation.cc:407] Connection header received
0928 13:02:44.962206 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:44.962207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:44.962221 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:44.962250 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:44.962652 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42288: BlockingRecv error: recv got EOF from 127.0.0.1:42288 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 13:02:45.031152 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:45.028892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.11.11.125:41219)
0928 13:02:45.028943 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.028946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.029217 (+ 271us) server_negotiation.cc:407] Connection header received
0928 13:02:45.029304 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.029306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.029320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.029404 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:45.031092 (+ 1688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42300: BlockingRecv error: recv got EOF from 127.0.0.1:42300 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 13:02:45.160974 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:45.160146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.11.11.125:41219)
0928 13:02:45.160225 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.160227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.160343 (+ 116us) server_negotiation.cc:407] Connection header received
0928 13:02:45.160425 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.160427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.160440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.160520 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:45.160891 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42308: BlockingRecv error: recv got EOF from 127.0.0.1:42308 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:02:45.358990 1417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "789b5d2f5615452d85e6e829182d3c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "789b5d2f5615452d85e6e829182d3c40" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 41883 } } peers { permanent_uuid: "b4120b2d0047410486ef9432783370d4" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 41219 } } peers { permanent_uuid: "5131ab8ef944496ca1b906883b3e475f" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 40303 } } }
I20260928 13:02:45.359179 1417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789b5d2f5615452d85e6e829182d3c40 [sys.catalog]: This master's current role is: LEADER
W20260928 13:02:45.419004 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:45.418218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.11.11.125:41219)
0928 13:02:45.418315 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.418317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.418346 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:02:45.418452 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.418453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.418471 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.418533 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:45.418906 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42318: BlockingRecv error: recv got EOF from 127.0.0.1:42318 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:02:45.788769 1308 catalog_manager.cc:1259] Loaded cluster ID: 89115d94dd954d03b74e5ea060ba4aab
I20260928 13:02:45.788849 1308 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: loading cluster ID for follower catalog manager: success
I20260928 13:02:45.789667 1308 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: acquiring CA information for follower catalog manager: success
I20260928 13:02:45.789990 1308 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5131ab8ef944496ca1b906883b3e475f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:02:45.858829 1355 catalog_manager.cc:1259] Loaded cluster ID: 89115d94dd954d03b74e5ea060ba4aab
I20260928 13:02:45.858913 1355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: loading cluster ID for follower catalog manager: success
I20260928 13:02:45.859879 1355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: acquiring CA information for follower catalog manager: success
I20260928 13:02:45.860246 1355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4120b2d0047410486ef9432783370d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:02:45.935273 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:45.932319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.11.11.125:41219)
0928 13:02:45.932446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.932448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.932493 (+ 45us) server_negotiation.cc:407] Connection header received
0928 13:02:45.932605 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.932606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.932666 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.932755 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:45.932934 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.933454 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.934398 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.934612 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.934690 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:45.934877 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:45.935006 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:45.935096 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:45.935138 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 13:02:45.935161 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260928 13:02:45.939922 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:45.937323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.11.11.124:40303)
0928 13:02:45.937446 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.937448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.937457 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:02:45.937579 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.937580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.937637 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.937726 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:45.937845 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.938338 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.939086 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.939307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.939392 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:45.939574 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:45.939693 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:45.939782 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:45.939806 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 13:02:45.939828 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260928 13:02:45.944819 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:45.941850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.11.11.125:41219)
0928 13:02:45.941904 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.941906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.942100 (+ 194us) server_negotiation.cc:407] Connection header received
0928 13:02:45.942166 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.942167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.942180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.942235 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:45.942412 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.942882 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.943977 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.944289 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.944359 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:45.944503 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:45.944540 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:45.944590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:45.944694 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 13:02:45.944721 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:02:45.945034 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:45.941757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.11.11.126:41883)
0928 13:02:45.941886 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.941888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.942056 (+ 168us) server_negotiation.cc:407] Connection header received
0928 13:02:45.942179 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.942180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.942235 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.942309 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:45.942519 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.943243 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.944014 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.944355 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.944394 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:45.944625 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:45.944741 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:45.944803 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:45.944902 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 13:02:45.944928 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 13:02:45.945144 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:45.941947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.11.11.124:40303)
0928 13:02:45.941984 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:45.941985 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:45.941994 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:02:45.942096 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:45.942097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:45.942107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:45.942153 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:45.942339 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.943090 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.944120 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:45.944360 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:45.944425 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:45.944573 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:45.944606 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:45.944654 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:45.944788 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 13:02:45.944810 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:02:45.946672 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1072
I20260928 13:02:45.952174 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1147
I20260928 13:02:45.957635 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1223
2026-09-28T13:02:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2806 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:02:46 dist-test-slave-8sjz krb5kdc[1448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:02:46 dist-test-slave-8sjz krb5kdc[1448](info): set up 2 sockets
Sep 28 13:02:46 dist-test-slave-8sjz krb5kdc[1448](info): commencing operation
krb5kdc: starting...
W20260928 13:02:46.993443 11309 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/test-user.keytab.
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.11.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
Entry for principal kudu/127.11.11.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab.
I20260928 13:02:47.121250 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46483
--webserver_interface=127.11.11.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:02:47.202525 1461 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:47.202749 1461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:47.202765 1461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:47.202785 1461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:47.203871 1461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:47.203905 1461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:47.203919 1461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:47.203933 1461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:47.203944 1461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:47.205369 1461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.126.keytab
--principal=kudu/127.11.11.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.126:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.11.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:47.205572 1461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:47.205744 1461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:47.210597 1466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:47.210641 1467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.210877 1461 server_base.cc:1034] running on GCE node
W20260928 13:02:47.210597 1469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.211179 1461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:47.211485 1461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:47.212677 1461 hybrid_clock.cc:648] HybridClock initialized: now 1790600567212660 us; error 47 us; skew 500 ppm
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:47.215330 1461 init.cc:377] Logged in from keytab as kudu/127.11.11.126@KRBTEST.COM (short username kudu)
I20260928 13:02:47.215819 1461 webserver.cc:460] Webserver started at http://127.11.11.126:46637/ using document root <none> and password file <none>
I20260928 13:02:47.216130 1461 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:47.216207 1461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:47.216370 1461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:47.217537 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data/instance:
uuid: "67148f26f7124862b3317344820b0f42"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "1d4ffd225a0b020e9d75d5572cd9373f"
server_key_iv: "a545508133272249078aaabda2c6764d"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.217968 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance:
uuid: "67148f26f7124862b3317344820b0f42"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "1d4ffd225a0b020e9d75d5572cd9373f"
server_key_iv: "a545508133272249078aaabda2c6764d"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.219352 1461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:02:47.220223 1476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:47.220398 1461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:47.220503 1461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal
uuid: "67148f26f7124862b3317344820b0f42"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "1d4ffd225a0b020e9d75d5572cd9373f"
server_key_iv: "a545508133272249078aaabda2c6764d"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.220587 1461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:47.236135 1461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:47.238612 1461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:47.238798 1461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:47.242766 1461 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.126:46483
I20260928 13:02:47.242806 1529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.126:46483 every 8 connection(s)
I20260928 13:02:47.243185 1461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/data/info.pb
I20260928 13:02:47.244270 1530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:47.245877 1530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.246479 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1461
I20260928 13:02:47.246557 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-0/wal/instance
I20260928 13:02:47.246778 11309 external_mini_cluster.cc:1428] Setting key 3765d70870212824b75fff7d06f31d15
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.125@KRBTEST.COM" created.
I20260928 13:02:47.258206 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.246720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46483 (local address 127.0.0.1:53504)
0928 13:02:47.246967 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.247018 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.247782 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.248108 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.248112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.248503 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.248670 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.248675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.250035 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.250036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.251253 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.251256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.251819 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.251821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.251914 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.252636 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.252648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.254314 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.256415 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.256422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.256424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.256634 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.256915 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.256916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.256917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.257012 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.257408 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.257413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.257477 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.257588 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.257817 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":78,"thread_start_us":113,"threads_started":1}
I20260928 13:02:47.258206 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.246747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.11.11.126:46483)
0928 13:02:47.246953 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.246963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.247127 (+ 164us) server_negotiation.cc:407] Connection header received
0928 13:02:47.247941 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.247943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.247961 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.248120 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.248773 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.249932 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.251368 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.251701 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.251850 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.254430 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.254437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.254437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.254444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.256309 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.256737 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.256741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.256741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.256834 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.257081 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.257081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.257082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.257326 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.257418 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.257625 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.257754 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260928 13:02:47.259847 1530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.260290 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.260159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40018)
0928 13:02:47.260212 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.260262 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:02:47.260437 1480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
I20260928 13:02:47.260883 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.260782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40030)
0928 13:02:47.260828 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.260852 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:02:47.260983 1530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
Entry for principal kudu/127.11.11.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab.
I20260928 13:02:47.275831 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:36181
--webserver_interface=127.11.11.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:02:47.310230 1530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } attempt: 1
I20260928 13:02:47.310773 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.310630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40038)
0928 13:02:47.310696 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.310738 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 13:02:47.311169 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.311070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40042)
0928 13:02:47.311119 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.311145 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:02:47.311259 1530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
W20260928 13:02:47.359169 1537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:47.359400 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:47.359419 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:47.359441 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:47.360716 1537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:47.360760 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:47.360774 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:47.360787 1537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:47.360800 1537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:47.362345 1537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.125.keytab
--principal=kudu/127.11.11.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.125:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.11.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:47.362623 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:47.362872 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:47.367818 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:47.367810 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.367882 1537 server_base.cc:1034] running on GCE node
W20260928 13:02:47.367859 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.368281 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:47.368588 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:47.369761 1537 hybrid_clock.cc:648] HybridClock initialized: now 1790600567369745 us; error 40 us; skew 500 ppm
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:47.372346 1537 init.cc:377] Logged in from keytab as kudu/127.11.11.125@KRBTEST.COM (short username kudu)
I20260928 13:02:47.372816 1537 webserver.cc:460] Webserver started at http://127.11.11.125:40873/ using document root <none> and password file <none>
I20260928 13:02:47.373070 1537 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:47.373118 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:47.373255 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:47.374317 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data/instance:
uuid: "331e2bc0509e436eb4b933ddee5c8d2a"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "a95e25db95e9e778c7cf628685eb66a6"
server_key_iv: "248aeacc5abd3e284f164f40478260cf"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.374678 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal/instance:
uuid: "331e2bc0509e436eb4b933ddee5c8d2a"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "a95e25db95e9e778c7cf628685eb66a6"
server_key_iv: "248aeacc5abd3e284f164f40478260cf"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.376156 1537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:02:47.377028 1552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:47.377231 1537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:02:47.377337 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal
uuid: "331e2bc0509e436eb4b933ddee5c8d2a"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "a95e25db95e9e778c7cf628685eb66a6"
server_key_iv: "248aeacc5abd3e284f164f40478260cf"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.377414 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:47.379410 1530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } attempt: 2
I20260928 13:02:47.379848 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.379715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40054)
0928 13:02:47.379776 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.379817 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:02:47.380224 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.380127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40062)
0928 13:02:47.380175 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.380200 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:02:47.380317 1530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.125:36181: Network error: Client connection negotiation failed: client connection to 127.11.11.125:36181: connect: Connection refused (error 111)
I20260928 13:02:47.399968 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:47.402329 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:47.402534 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:47.406702 1537 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.125:36181
I20260928 13:02:47.406733 1604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.125:36181 every 8 connection(s)
I20260928 13:02:47.407159 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/data/info.pb
I20260928 13:02:47.408215 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:47.409749 1605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.411128 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1537
I20260928 13:02:47.411237 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-1/wal/instance
I20260928 13:02:47.411453 11309 external_mini_cluster.cc:1428] Setting key 83740ff1bfc3cd52ede548acafc14c8c
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.11.124@KRBTEST.COM; defaulting to no policy
I20260928 13:02:47.420697 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.410500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.11.11.126:46483)
0928 13:02:47.410587 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.410589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.410788 (+ 199us) server_negotiation.cc:407] Connection header received
0928 13:02:47.411520 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.411522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.411540 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.411620 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.412598 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.413136 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.414508 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.414724 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.414786 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.417281 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.417283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.417283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.417292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.419012 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.419495 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.419497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.419497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.419564 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.419893 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.419895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.419895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.420098 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.420176 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.420543 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.420622 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 13:02:47.421033 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.410453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46483 (local address 127.0.0.1:53514)
0928 13:02:47.410639 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.410684 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.411352 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.411724 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.411728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.412156 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.412456 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.412462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.413276 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.413277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.414414 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.414417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.414842 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.414844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.415006 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.415555 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.415568 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.417175 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.419120 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.419129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.419131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.419403 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.419662 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.419663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.419663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.419797 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.420182 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.420190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.420352 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.420478 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.420644 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260928 13:02:47.421793 1605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.11.124@KRBTEST.COM" created.
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:47.430347 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.422081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40072)
0928 13:02:47.422138 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.422148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.422190 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.422492 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.422493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.422642 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.422717 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.422719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.423795 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.423796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.424483 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.424488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.425185 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.425186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.425254 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.425741 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.425745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.427278 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.429190 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.429191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.429191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.429366 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.429592 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.429593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.429594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.429650 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.429977 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.429978 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.430018 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.430090 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.430206 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:02:47.430382 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.422152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.11.11.125:36181)
0928 13:02:47.422338 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.422342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.422366 (+ 24us) server_negotiation.cc:407] Connection header received
0928 13:02:47.422392 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.422394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.422405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.422492 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.422789 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.423667 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.424564 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.425090 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.425173 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.427382 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.427394 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.427394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.427402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.429079 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.429459 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.429463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.429463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.429525 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.429723 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.429724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.429724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.429902 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.430003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.430122 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.430268 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260928 13:02:47.431241 1605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.431619 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.431491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45124)
0928 13:02:47.431545 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.431593 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 13:02:47.431800 1553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
I20260928 13:02:47.432353 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.432259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45132)
0928 13:02:47.432307 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.432334 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:02:47.432464 1605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:41847: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
Entry for principal kudu/127.11.11.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab.
I20260928 13:02:47.439369 11309 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNYzOw4/build/release/bin/kudu
/tmp/dist-test-taskNYzOw4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:41847
--webserver_interface=127.11.11.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.11.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:02:47.471489 1530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } attempt: 3
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:47.478940 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.471958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40074)
0928 13:02:47.472027 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.472073 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.472134 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.472309 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.472310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.472482 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.472554 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.472556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.473257 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.473258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.473794 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.473795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.474209 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.474210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.474294 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.474787 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.474798 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.476094 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.477963 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.477964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.477964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.478136 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.478342 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.478342 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.478342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.478410 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.478683 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.478684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.478729 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.478783 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.478865 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 13:02:47.478998 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.472015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.11.11.125:36181)
0928 13:02:47.472088 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.472090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.472114 (+ 24us) server_negotiation.cc:407] Connection header received
0928 13:02:47.472218 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.472220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.472233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.472303 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.472649 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.473164 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.473893 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.474126 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.474211 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.476196 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.476198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.476199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.476209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.477876 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.478209 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.478210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.478211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.478275 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.478472 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.478473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.478473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.478619 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.478686 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.478826 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.478908 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:02:47.479427 1530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.479769 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.479668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45144)
0928 13:02:47.479715 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.479747 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:02:47.480126 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.480001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45160)
0928 13:02:47.480079 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.480110 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 13:02:47.480239 1530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:41847: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
I20260928 13:02:47.481660 1605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } attempt: 1
I20260928 13:02:47.482012 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.481900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45164)
0928 13:02:47.481952 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.481979 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:02:47.482304 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.482211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45174)
0928 13:02:47.482261 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.482283 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 13:02:47.482405 1605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:41847: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
I20260928 13:02:47.511440 1530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } attempt: 1
I20260928 13:02:47.512069 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.511873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45176)
0928 13:02:47.511951 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.511998 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 13:02:47.512463 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.512362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45186)
0928 13:02:47.512412 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.512440 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:02:47.512591 1530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:41847: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
W20260928 13:02:47.525444 1612 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:02:47.525722 1612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:02:47.525764 1612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:02:47.525810 1612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:02:47.527048 1612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:02:47.527118 1612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:02:47.527145 1612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:02:47.527177 1612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:02:47.527204 1612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:02:47.528853 1612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.11.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.11.126:46483,127.11.11.125:36181,127.11.11.124:41847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/krb5kdc/kudu_127.11.11.124.keytab
--principal=kudu/127.11.11.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.11.124:41847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.11.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:02:47.529152 1612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:02:47.529390 1612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:02:47.534298 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.534384 1612 server_base.cc:1034] running on GCE node
W20260928 13:02:47.534293 1620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:02:47.534293 1617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:02:47.534759 1612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:02:47.535071 1612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:02:47.536248 1612 hybrid_clock.cc:648] HybridClock initialized: now 1790600567536237 us; error 38 us; skew 500 ppm
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:02:47.539335 1612 init.cc:377] Logged in from keytab as kudu/127.11.11.124@KRBTEST.COM (short username kudu)
I20260928 13:02:47.539803 1612 webserver.cc:460] Webserver started at http://127.11.11.124:35113/ using document root <none> and password file <none>
I20260928 13:02:47.540124 1612 fs_manager.cc:359] Metadata directory not provided
I20260928 13:02:47.540220 1612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:02:47.540364 1612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:02:47.541462 1612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data/instance:
uuid: "d0f9b94c6bea4a0b9b4100770c9d299c"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "c66475f680bcae490f42ba821786842f"
server_key_iv: "e140375b3d960e9fb468d337509906f2"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.541842 1612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal/instance:
uuid: "d0f9b94c6bea4a0b9b4100770c9d299c"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "c66475f680bcae490f42ba821786842f"
server_key_iv: "e140375b3d960e9fb468d337509906f2"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.543279 1612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:02:47.544126 1627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:02:47.544309 1612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:02:47.544411 1612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data,/tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal
uuid: "d0f9b94c6bea4a0b9b4100770c9d299c"
format_stamp: "Formatted at 2026-09-28 13:02:47 on dist-test-slave-8sjz"
server_key: "c66475f680bcae490f42ba821786842f"
server_key_iv: "e140375b3d960e9fb468d337509906f2"
server_key_version: "encryptionkey@0"
I20260928 13:02:47.544497 1612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:02:47.550551 1605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } attempt: 2
I20260928 13:02:47.551056 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.550922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45192)
0928 13:02:47.550982 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.551025 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:02:47.551410 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.551317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45208)
0928 13:02:47.551361 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.551387 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:02:47.551530 1605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.11.124:41847: Network error: Client connection negotiation failed: client connection to 127.11.11.124:41847: connect: Connection refused (error 111)
I20260928 13:02:47.559978 1612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:02:47.562409 1612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:02:47.562528 1612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:02:47.566537 1612 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.124:41847
I20260928 13:02:47.566574 1680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.11.124:41847 every 8 connection(s)
I20260928 13:02:47.566947 1612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/data/info.pb
I20260928 13:02:47.567915 1681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:02:47.569509 1681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.574455 11309 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNYzOw4/build/release/bin/kudu as pid 1612
I20260928 13:02:47.574573 11309 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNYzOw4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600290352246-11309-0/minicluster-data/master-2/wal/instance
I20260928 13:02:47.574807 11309 external_mini_cluster.cc:1428] Setting key ec4e5fdcaa968463256890a83dacae05
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
I20260928 13:02:47.579185 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.570277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.11.11.126:46483)
0928 13:02:47.570356 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.570358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.570578 (+ 220us) server_negotiation.cc:407] Connection header received
0928 13:02:47.571021 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.571022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.571032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.571096 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.571704 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.572285 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.573249 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.573432 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.573486 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.576064 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.576066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.576066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.576075 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.577653 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.578055 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.578057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.578057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.578125 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.578441 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.578442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.578443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.578569 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.578622 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.578999 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.579076 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:02:47.579380 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.570233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.126:46483 (local address 127.0.0.1:53520)
0928 13:02:47.570435 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.570487 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.570940 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.571172 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.571175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.571423 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.571630 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.571634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.572362 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.572363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.573161 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.573164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.573527 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.573528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.573628 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.574105 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.574113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.575950 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.577736 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.577741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.577743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.577976 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.578222 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.578223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.578223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.578353 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.578636 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.578641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.578805 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.578933 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.579120 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
I20260928 13:02:47.579927 1681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:47.584728 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.575223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.11.11.124:41847)
0928 13:02:47.575452 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.575457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.575485 (+ 28us) server_negotiation.cc:407] Connection header received
0928 13:02:47.575513 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.575515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.575536 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.575619 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.576342 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.577265 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.578222 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.578675 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.578740 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.581471 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.581480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.581480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.581487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.583163 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.583547 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.583550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.583551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.583613 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.583827 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.583828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.583828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.583983 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.584164 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.584516 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.584599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":135,"threads_started":1}
I20260928 13:02:47.586555 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.580312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.11.11.125:36181)
0928 13:02:47.580358 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.580375 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.580382 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:02:47.580408 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.580409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.580420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.580468 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.580763 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.581282 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.582001 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.582199 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.582288 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.583894 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.583895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.583896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.583901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.585406 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.585810 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.585812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.585812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.585875 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.586080 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.586081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.586081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.586177 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.586223 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.586391 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.586449 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:47.586566 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.580219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.125:36181 (local address 127.0.0.1:40088)
0928 13:02:47.580274 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.580285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.580326 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.580490 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.580491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.580640 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.580684 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.580685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.581359 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.581360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.581917 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.581918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.582288 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.582289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.582366 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.582699 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.582703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.583787 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.585508 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.585509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.585510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.585675 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.585944 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.585945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.585945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.586014 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.586240 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.586242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.586293 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.586377 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.586461 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:02:47.586984 1681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:02:47.587729 1530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } attempt: 2
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.126@KRBTEST.COM
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.124@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.126@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:02:47.594696 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.587241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45226)
0928 13:02:47.587288 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.587298 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.587333 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.587455 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.587455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.587595 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.587628 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.587629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.588766 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.588767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.589398 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.589400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.589855 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.589856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.589902 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.590371 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.590376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.591792 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.593396 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.593397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.593397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.593557 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.593895 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.593896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.593896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.593975 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.594268 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.594269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.594306 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.594408 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.594562 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:02:47.595049 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.585762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.11.11.126:46483)
0928 13:02:47.585806 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.585807 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.585829 (+ 22us) server_negotiation.cc:407] Connection header received
0928 13:02:47.585852 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.585853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.585863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.585922 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.586527 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.587312 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.588342 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.588641 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.588715 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.591184 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.591185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.591185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.591192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.593352 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.593769 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.593770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.593771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.593822 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.594083 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.594084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.594085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.594212 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.594403 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.594849 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.594925 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:02:47.595474 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.587297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.11.11.124:41847)
0928 13:02:47.587332 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.587334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.587342 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:02:47.587389 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.587390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.587400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.587451 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.587694 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.588254 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.589499 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.589755 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.589814 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.591855 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.591857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.591857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.591871 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.593295 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.593641 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.593642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.593642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.593694 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.594047 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.594048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.594048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.594193 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.594265 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.594434 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.594639 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":17}
I20260928 13:02:47.597133 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.587989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45234)
0928 13:02:47.588057 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.588066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.588098 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.588458 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.588459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.588613 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.588647 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.588648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.589728 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.589729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.590533 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.590536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.591043 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.591043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.591100 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.591511 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.591515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.593103 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.595957 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.595958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.595958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.596200 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.596423 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.596424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.596425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.596484 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.596798 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.596799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.596850 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.596916 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.597010 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:02:47.597388 1681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: Bootstrap starting.
I20260928 13:02:47.597503 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.588167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.11.11.124:41847)
0928 13:02:47.588334 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.588336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.588343 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:02:47.588361 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.588362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.588370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.588437 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.588898 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.589630 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.590626 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.590964 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.591042 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.593598 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.593600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.593600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.593607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.595780 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.596289 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.596290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.596290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.596346 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.596563 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.596564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.596564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.596736 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.596804 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.596975 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.597053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 13:02:47.598204 1681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:47.598510 1681 log.cc:824] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:47.599048 1530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: Bootstrap starting.
I20260928 13:02:47.599416 1681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: No bootstrap required, opened a new log
I20260928 13:02:47.599781 1530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:47.600113 1530 log.cc:824] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:47.600824 1530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: No bootstrap required, opened a new log
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.11.125@KRBTEST.COM
I20260928 13:02:47.601787 1681 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.601954 1681 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f9b94c6bea4a0b9b4100770c9d299c, State: Initialized, Role: FOLLOWER
I20260928 13:02:47.602056 1681 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.602191 1530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.602353 1530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67148f26f7124862b3317344820b0f42, State: Initialized, Role: FOLLOWER
I20260928 13:02:47.602380 1689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.602448 1530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.602505 1689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.602392 1681 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:47.602785 1530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:47.604221 1690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.604316 1690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:02:47.604971 1712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:47.605022 1712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:47.606210 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.595792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.11.11.125:36181)
0928 13:02:47.595860 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.595862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.595920 (+ 58us) server_negotiation.cc:407] Connection header received
0928 13:02:47.596022 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.596024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.596057 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.596105 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.596731 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.597511 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.598755 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.598994 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.599061 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.602628 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.602630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.602630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.602640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.604651 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.605074 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.605075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.605076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.605149 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.605360 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.605361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.605362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.605508 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.605628 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.606027 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.606109 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:02:47.605799 1461 master_runner.cc:186] Error getting master registration for 127.11.11.125:36181: OK, instance_id { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" instance_seqno: 1790600567405500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:02:47.607748 1461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 13:02:47.608198 1708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:47.608256 1708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:47.608898 11309 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:02:47.606477 1612 master_runner.cc:186] Error getting master registration for 127.11.11.125:36181: OK, instance_id { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" instance_seqno: 1790600567405500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:02:47.609532 1612 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:02:47.609782 1504 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "67148f26f7124862b3317344820b0f42"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53532
I20260928 13:02:47.609894 1504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 13:02:47.609951 1504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:47.610930 1504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.611161 1504 leader_election.cc:288] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [CANDIDATE]: Term 1 election: Requested vote from peers 331e2bc0509e436eb4b933ddee5c8d2a (127.11.11.125:36181), d0f9b94c6bea4a0b9b4100770c9d299c (127.11.11.124:41847)
I20260928 13:02:47.612449 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67148f26f7124862b3317344820b0f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c"
I20260928 13:02:47.612571 1655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:02:47.613497 1655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67148f26f7124862b3317344820b0f42 in term 1.
I20260928 13:02:47.613737 1477 leader_election.cc:302] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67148f26f7124862b3317344820b0f42, d0f9b94c6bea4a0b9b4100770c9d299c; no voters:
I20260928 13:02:47.613878 1690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:02:47.614039 1690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [term 1 LEADER]: Becoming Leader. State: Replica: 67148f26f7124862b3317344820b0f42, State: Running, Role: LEADER
I20260928 13:02:47.614166 1690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.614640 1690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67148f26f7124862b3317344820b0f42. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.614768 1690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:47.620541 1724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:02:47.620820 1724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:02:47.623781 1655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [term 1 FOLLOWER]: Refusing update from remote peer 67148f26f7124862b3317344820b0f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:47.625180 1721 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:47.626631 1689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67148f26f7124862b3317344820b0f42. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.626726 1689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.628806 1725 mvcc.cc:205] Tried to move back new op lower bound from 7334299924980121600 to 7334299924948537344. Current Snapshot: MvccSnapshot[applied={T|T < 7334299924980121600}]
I20260928 13:02:47.629374 1690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.629463 1690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:47.629997 1689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.630090 1689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.636698 1724 catalog_manager.cc:1347] Generated new cluster ID: d61654d2294d4dd5a1abe3fb45e05658
I20260928 13:02:47.636787 1724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:02:47.636770 1726 mvcc.cc:205] Tried to move back new op lower bound from 7334299924980121600 to 7334299924948537344. Current Snapshot: MvccSnapshot[applied={T|T < 7334299924980121600}]
I20260928 13:02:47.637089 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.612512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.11.11.126:46483)
0928 13:02:47.612562 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.612565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.620380 (+ 7815us) server_negotiation.cc:407] Connection header received
0928 13:02:47.620427 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.620428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.620442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.620518 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.622430 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.623210 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.625131 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.625474 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.625545 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.631286 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.631288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.631288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.631297 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.633592 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.634403 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.634404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.634405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.634467 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.634662 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.634663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.634663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.634796 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.634891 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.636891 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.636997 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:02:47.638397 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.612809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.11.11.124:41847)
0928 13:02:47.613613 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.613615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.626913 (+ 13298us) server_negotiation.cc:407] Connection header received
0928 13:02:47.627028 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.627029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.627040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.627069 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.628294 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.629058 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.631739 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.632098 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.632141 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.634107 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.634108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.634108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.634118 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.635954 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.637065 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.637067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.637067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.637129 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.637700 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.637701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.637701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.637827 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.637919 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.638213 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.638299 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782}
I20260928 13:02:47.642659 1605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } attempt: 3
I20260928 13:02:47.645438 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.613910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.11.11.125:36181)
0928 13:02:47.613953 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.613955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.631688 (+ 17733us) server_negotiation.cc:407] Connection header received
0928 13:02:47.632552 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.632554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.632567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.632599 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.635097 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.635839 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.637813 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.638616 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.638666 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.640903 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.640904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.640905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.640913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.642512 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.644519 (+ 2007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.644520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.644521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.644600 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.644803 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.644804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.644804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.644943 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.645041 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.645245 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.645337 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 13:02:47 dist-test-slave-8sjz krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600567, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.11.125@KRBTEST.COM for kudu/127.11.11.124@KRBTEST.COM
I20260928 13:02:47.651319 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.642941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.11.124:41847 (local address 127.0.0.1:45254)
0928 13:02:47.643008 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:02:47.643018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:02:47.643055 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:02:47.643209 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:02:47.643210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:02:47.643365 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:02:47.643414 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.643416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.644420 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.644421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.645261 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:02:47.645264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.646207 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.646209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:02:47.646292 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.646784 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:02:47.646789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:02:47.648385 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:02:47.650181 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.650182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.650183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.650407 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.650636 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:02:47.650637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:02:47.650637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:02:47.650704 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:02:47.650987 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:02:47.650988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:02:47.651038 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 13:02:47.651110 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:02:47.651213 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 13:02:47.651836 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:47.643002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.11.11.124:41847)
0928 13:02:47.643053 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.643054 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.643063 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:02:47.643122 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.643123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.643133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.643203 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.643505 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.644316 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.645393 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:47.645749 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:47.645820 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:47.648479 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:02:47.648481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:02:47.648481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:02:47.648490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:02:47.650092 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.650485 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.650486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.650486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.650564 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:02:47.650773 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:02:47.650774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:02:47.650774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:02:47.650911 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:02:47.650976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:47.651165 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 13:02:47.651316 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 13:02:47.653858 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: Bootstrap starting.
I20260928 13:02:47.654670 1605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: Neither blocks nor log segments found. Creating new log.
I20260928 13:02:47.654968 1605 log.cc:824] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:02:47.655920 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: No bootstrap required, opened a new log
I20260928 13:02:47.657882 1605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.658015 1605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331e2bc0509e436eb4b933ddee5c8d2a, State: Initialized, Role: FOLLOWER
I20260928 13:02:47.658084 1605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } }
I20260928 13:02:47.658418 1605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:02:47.658387 1732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.658484 1732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.659847 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67148f26f7124862b3317344820b0f42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "331e2bc0509e436eb4b933ddee5c8d2a"
I20260928 13:02:47.659958 1579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [term 0 FOLLOWER]: Advancing to term 1
W20260928 13:02:47.660233 1744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:02:47.660284 1744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:02:47.661178 1579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67148f26f7124862b3317344820b0f42 in term 1.
I20260928 13:02:47.661934 1578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [term 1 FOLLOWER]: Refusing update from remote peer 67148f26f7124862b3317344820b0f42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:02:47.662226 1729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:02:47.663589 1732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67148f26f7124862b3317344820b0f42. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.663673 1732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.663707 1732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.663745 1732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:02:47.664851 1729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67148f26f7124862b3317344820b0f42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67148f26f7124862b3317344820b0f42" member_type: VOTER last_known_addr { host: "127.11.11.126" port: 46483 } } peers { permanent_uuid: "331e2bc0509e436eb4b933ddee5c8d2a" member_type: VOTER last_known_addr { host: "127.11.11.125" port: 36181 } } peers { permanent_uuid: "d0f9b94c6bea4a0b9b4100770c9d299c" member_type: VOTER last_known_addr { host: "127.11.11.124" port: 41847 } } }
I20260928 13:02:47.664947 1729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42 [sys.catalog]: This master's current role is: LEADER
I20260928 13:02:47.665387 1724 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:02:47.666141 1724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:02:47.666484 1746 mvcc.cc:205] Tried to move back new op lower bound from 7334299924980121600 to 7334299924948537344. Current Snapshot: MvccSnapshot[applied={T|T < 7334299924980121600}]
I20260928 13:02:47.669046 1724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67148f26f7124862b3317344820b0f42: Generated new TSK 0
I20260928 13:02:47.669232 1724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:02:47.696852 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.695676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.11.11.125:36181)
0928 13:02:47.695729 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.695732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.696123 (+ 391us) server_negotiation.cc:407] Connection header received
0928 13:02:47.696237 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.696238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.696250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.696282 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.696778 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40112: BlockingRecv error: recv got EOF from 127.0.0.1:40112 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 13:02:47.698311 1532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.697430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.11.11.126:46483)
0928 13:02:47.697472 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.697474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.697767 (+ 293us) server_negotiation.cc:407] Connection header received
0928 13:02:47.697866 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.697867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.697879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.697908 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.698242 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53546: BlockingRecv error: recv got EOF from 127.0.0.1:53546 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:02:47.699250 1686 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.697654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.11.11.124:41847)
0928 13:02:47.697694 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.697696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.698744 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 13:02:47.698839 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.698840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.698851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.698880 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.699185 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45258: BlockingRecv error: recv got EOF from 127.0.0.1:45258 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:02:47.701592 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.700873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.11.11.125:36181)
0928 13:02:47.700923 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.700925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.701037 (+ 112us) server_negotiation.cc:407] Connection header received
0928 13:02:47.701152 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.701153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.701174 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.701204 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.701530 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40124: BlockingRecv error: recv got EOF from 127.0.0.1:40124 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 13:02:47.704985 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.704172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.11.11.125:36181)
0928 13:02:47.704209 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.704211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.704476 (+ 265us) server_negotiation.cc:407] Connection header received
0928 13:02:47.704574 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.704575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.704586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.704616 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.704927 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40128: BlockingRecv error: recv got EOF from 127.0.0.1:40128 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 13:02:47.709167 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.11.11.125:36181)
0928 13:02:47.708350 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.708352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.708617 (+ 265us) server_negotiation.cc:407] Connection header received
0928 13:02:47.708721 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.708722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.708734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.708763 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.709111 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40138: BlockingRecv error: recv got EOF from 127.0.0.1:40138 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 13:02:47.715438 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.714553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.11.11.125:36181)
0928 13:02:47.714596 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.714598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.714866 (+ 268us) server_negotiation.cc:407] Connection header received
0928 13:02:47.714976 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.714977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.714988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.715019 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.715378 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40152: BlockingRecv error: recv got EOF from 127.0.0.1:40152 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:02:47.725988 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.724899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.11.11.125:36181)
0928 13:02:47.724946 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.724949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.725280 (+ 331us) server_negotiation.cc:407] Connection header received
0928 13:02:47.725416 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.725417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.725431 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.725464 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.725925 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40160: BlockingRecv error: recv got EOF from 127.0.0.1:40160 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:02:47.744666 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.743558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.11.11.125:36181)
0928 13:02:47.743608 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.743610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.743949 (+ 339us) server_negotiation.cc:407] Connection header received
0928 13:02:47.744107 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.744109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.744122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.744157 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.744602 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40168: BlockingRecv error: recv got EOF from 127.0.0.1:40168 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 13:02:47.779349 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.778292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.11.11.125:36181)
0928 13:02:47.778345 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.778347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.778663 (+ 316us) server_negotiation.cc:407] Connection header received
0928 13:02:47.778776 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.778777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.778790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.778820 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.779285 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40172: BlockingRecv error: recv got EOF from 127.0.0.1:40172 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 13:02:47.845821 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.844990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.11.11.125:36181)
0928 13:02:47.845065 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.845067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.845192 (+ 125us) server_negotiation.cc:407] Connection header received
0928 13:02:47.845245 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.845246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.845256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.845315 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.845754 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40184: BlockingRecv error: recv got EOF from 127.0.0.1:40184 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 13:02:47.975903 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:47.975029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.11.11.125:36181)
0928 13:02:47.975105 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:47.975108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:47.975216 (+ 108us) server_negotiation.cc:407] Connection header received
0928 13:02:47.975263 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:47.975264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:47.975275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:47.975329 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:47.975734 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40186: BlockingRecv error: recv got EOF from 127.0.0.1:40186 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 13:02:48.233842 1609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:02:48.233086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.11.11.125:36181)
0928 13:02:48.233160 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.233162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.233206 (+ 44us) server_negotiation.cc:407] Connection header received
0928 13:02:48.233301 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.233302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.233314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.233391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:02:48.233779 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40196: BlockingRecv error: recv got EOF from 127.0.0.1:40196 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 13:02:48.608700 1708 catalog_manager.cc:1259] Loaded cluster ID: d61654d2294d4dd5a1abe3fb45e05658
I20260928 13:02:48.608780 1708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: loading cluster ID for follower catalog manager: success
I20260928 13:02:48.609659 1708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: acquiring CA information for follower catalog manager: success
I20260928 13:02:48.610002 1708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0f9b94c6bea4a0b9b4100770c9d299c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:02:48.660730 1744 catalog_manager.cc:1259] Loaded cluster ID: d61654d2294d4dd5a1abe3fb45e05658
I20260928 13:02:48.660861 1744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: loading cluster ID for follower catalog manager: success
I20260928 13:02:48.661727 1744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: acquiring CA information for follower catalog manager: success
I20260928 13:02:48.662070 1744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 331e2bc0509e436eb4b933ddee5c8d2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:02:48.749768 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:48.747172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.11.11.125:36181)
0928 13:02:48.747302 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.747304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.747361 (+ 57us) server_negotiation.cc:407] Connection header received
0928 13:02:48.747388 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.747389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.747440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.747505 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:48.747662 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.748250 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.749023 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.749216 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.749271 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:48.749427 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:48.749536 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:48.749591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:48.749631 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 13:02:48.749648 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260928 13:02:48.754269 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:48.751728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.11.11.124:41847)
0928 13:02:48.751871 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.751873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.751884 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:02:48.751980 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.751981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.752020 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.752156 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:48.752246 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.752764 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.753518 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.753716 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.753774 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:48.753913 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:48.754021 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:48.754080 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:48.754119 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 13:02:48.754138 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260928 13:02:48.759666 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:48.756580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.11.11.124:41847)
0928 13:02:48.756627 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.756628 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.756701 (+ 73us) server_negotiation.cc:407] Connection header received
0928 13:02:48.756763 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.756764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.756774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.756817 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:48.757083 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.757850 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.758777 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.759103 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.759161 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:48.759343 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:48.759401 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:48.759458 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:48.759537 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 13:02:48.759561 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:02:48.759867 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:48.756373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.11.11.125:36181)
0928 13:02:48.756417 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.756420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.756574 (+ 154us) server_negotiation.cc:407] Connection header received
0928 13:02:48.756691 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.756692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.756706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.756758 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:48.757176 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.757904 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.758947 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.759272 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.759354 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:48.759579 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:48.759615 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:48.759643 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:48.759752 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 13:02:48.759781 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:02:48.760066 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:02:48.756341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.11.11.126:46483)
0928 13:02:48.756456 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 13:02:48.756458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:02:48.756596 (+ 138us) server_negotiation.cc:407] Connection header received
0928 13:02:48.756651 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:02:48.756652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:02:48.756712 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:02:48.756789 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:02:48.757102 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.757656 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.758733 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:02:48.759086 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:02:48.759196 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:02:48.759547 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:02:48.759708 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:02:48.759766 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:02:48.759925 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 13:02:48.759953 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260928 13:02:48.766607 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1461
I20260928 13:02:48.772250 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1537
I20260928 13:02:48.777385 11309 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNYzOw4/build/release/bin/kudu with pid 1612
2026-09-28T13:02:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2818 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5625 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278431 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST